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Overview 


One of the original purposes of student financial aid was to ensure fairer access to postsecondary 
education to those least able to afford it and to those traditionally underrepresented. Various federal 
and state programs were put in place to achieve this goal, including the federal Pell Grant and state 
aid programs. Y et policymakers and education leaders continue to grapple with how to boost college 
attendance and completion in an era when the number of college graduates is lagging behind de- 
mand and government resources are increasingly limited. Increasing the number of college gradu- 
ates is particularly difficult given the continued rising cost of attending college and the failure of 
financial aid to keep pace. 

This report presents early findings from an evaluation of performance-based scholarships targeting 
college-bound high school seniors in California, referred to as the Cash for College Performance- 
Based Scholarship (CFC-PBS) Program. This program is one of six being studied as part of the Per- 
formance-Based Scholarship (PBS) Demonstration. Performance-based scholarships are need-based 
grants contingent on meeting certain academic benchmarks to receive payment — in this case, a 
half-time course load with a “C” or better grade point average (GPA). Unlike merit-based scholar- 
ships, there are no academic criteria to be eligible for the program at the outset. The CFC-PBS 
scholarship can be taken to any accredited, degree-granting college or university in the country. The 
goal of the CFC-PBS Program is to increase the amount of aid available for students while simulta- 
neously providing an incentive for academic achievement. 

Using a random assignment design — the gold-standard methodology in program evaluation — 
MDRC assigned over 5,000 students to one of five program groups that were eligible for the incentive 
scholarship, to a group that was eligible for a scholarship without performance criteria, or to a control 
group that received their colleges’ standard financial aid packages. This report analyzes three terms of 
follow-up data from the program in California. Overall, the findings in this report show the following: 

• The CFC-PBS Program was largely implemented as designed. 

• While few students received the entire amount of the scholarship for which they were eligi- 
ble, most students received some funding. In this way, the design of the scholarship enabled 
more students to receive additional financial aid. 

• The CFC-PBS Program encouraged more students to matriculate, by about 5 percentage 
points above the control group rate of 84.4 percent. This increased matriculation largely oc- 
curred at community colleges. Flowever, the program had only limited effects on persis- 
tence from semester to semester, and only for community college students. 

• The program had positive impacts on academic success. These effects extend to numerous 
subgroups, such as males, females, and students of Latino ethnicity. There is strong evi- 
dence that the program affected students with lower high school GPAs more than students 
with higher high school GPAs. 

• The cost to administer scholarships increased as performance requirements were added, but 
since on the whole, students received only a portion of the scholarship amount they were of- 
fered, the decrease in payments to students more than offset the increased cost of admin- 
istration. All else being equal, scholarships with more performance requirements cost less 
than scholarships with fewer performance requirements. 

A future report will present a cross-site synthesis of the final results from this and other sites from 
the PBS Demonstration programs. 

iii 




Contents 


Overview iii 

List of Exhibits vii 

Preface xi 

Acknowledgments xiii 

Executive Summary ES-1 

Chapter 

1 Introduction 1 

A Brief Overview of the PBS Demonstration 3 

History and Structure of the Cash for College Performance-Based Scholarship Program 8 

The Higher Education Landscape in California 12 

Theory of Change 14 

Organization of the Report 1 7 

2 Sample Intake, Sample Characteristics, Data Sources, and Methodology 19 

Target Population 19 

Recruitment and Random Assignment 20 

Data Sources 25 

Methodology 28 

Characteristics of the CFC-PBS Sample 29 

3 Program Implementation 41 

Program Operations: Embedding Performance-Based Scholarships into Cash for College 42 
Scholarship Payments 46 

Was the Program Implemented as Designed? 49 

Student Perceptions and Experiences 54 

Conclusion 60 

4 Overall Effects on Academic and Other Outcomes 63 

Early Academic Outcomes 64 

Subgroup Analyses of the Effect of Performance-Based Scholarships 74 

Understanding the Effects of the Scholarship 77 

Conclusion 89 


v 



5 Cost Analysis of the Cash for College Performance-Based 


Scholarship Program 93 

Key Findings 93 

Methodology 95 

Cost Analysis 96 

Conclusion 104 

6 Policy Implications and Conclusions 105 

Placing the Findings in the Context of the Research on Incentives 106 

Implications for Innovations in Financial Aid Policy 116 

Appendix 

A Technical Appendix 119 

B Supplementary Exhibits for Chapter 2 133 

C Supplementary Exhibit for Chapter 3 139 

D Supplementary Exhibits for Chapter 4 143 

E Supplementary Information and Exhibits for Chapter 5 155 

F Supplementary Exhibits for Chapter 6 163 

References 171 

Earlier MDRC Publications on the Performance-Based 

Scholarship Demonstration 175 


vi 



List of Exhibits 


Table 

1 . 1 Summary of MDRC Performance-Based Scholarship Interventions 5 

2. 1 Selected Characteristics of Sample Members and All 2010 Cash for College 

Attendees at Baseline, by Region 30 

2.2 Selected Characteristics of Sample Members at Baseline, by Research Group 35 

2.3 Financial Aid Application Characteristics of Sample Members at Baseline 37 

3.1 Semester Payment Schedule 44 

3.2 Scholarship Receipt Among Program Group Members: First Through Third 

Terms 51 

3.3 PBS Survey Responses: Program Group Experiences 57 

3.4 PBS Survey Responses: Scholarship Use Among the Program Group 59 

4. 1 Enrollment Outcomes: First Through Third Terms 66 

4.2 Enrollment Impact Over the Control Group, by Scholarship Type: First Through 

Third Terms 72 

4.3 Enrollment by Subgroups: First Term 75 

4.4 Enrollment by Subgroups: Second Term 76 

4.5 Enrollment by Subgroups: Third Term 78 

4.6 Academic Outcomes in California Community Colleges: First Through 

Second Terms 79 

4.7 Enrollment Outcomes, by Cal Grant A or B Eligibility Prior to Random As- 
signment: First Through Third Terms 83 

4.8 PBS Survey Responses: Social Support, Flealth, and Motivation 86 

4.9 PBS Survey Responses: Educational Experiences 88 

4.10 PBS Survey Responses: Employment 90 

6. 1 Postsecondary Outcomes Across Prior Research on Financial Incentives 108 

A. 1 Selected Characteristics of Sample Members at Baseline, Based on Whether 

Found by the Clearinghouse 125 

vii 



A.2 Selected Characteristics of PBS Survey Respondents at Baseline, by Research 



Group 

128 

A.3 

Selected Characteristics of Sample Members at Baseline, by PBS Survey 
Respondents and Nonrespondents 

130 

C.l 

Quarter Payment Schedule 

141 

D.l 

Impact Matrix of Enrollment in First Term, by Scholarship Type 

145 

D.2 

Impact Matrix of Enrollment in Second Term, by Scholarship Type 

147 

D.3 

Enrollment Outcomes Among Students with One-Term Scholarships, Non- 
Performance-Based Versus Performance-Based: First Through Third Terms 

149 

D.4 

Enrollment Outcomes Among Students with One-Year Scholarships, $500 per 
Temi Versus $1,000 per Term: First Through Third Terms 

151 

D.5 

Enrollment Outcomes Among Students with Two-Year Scholarships, $500 per 
Term Versus $1,000 per Term: First Through Third Terms 

153 

E.l 

Examining Marginal Cost Estimates 

158 

F.l 

Descriptions of Prior External Incentive Research 

165 

Figure 



ES.l 

Average Scholarship Amount Received Among Program Group Members: First 
Through Third Terms 

ES-6 

ES.2 

Matriculation 

ES-9 

1.1 

Cash for College Workshop Locations 

10 

1.2 

Logic Model for the Cash for College Performance -Based Scholarship 

16 

2.1 

Random Assignment Diagram 

24 

5.1 

Cost per Program Group Member, by Scholarship Type, Over the Life of the 
Scholarship 

98 

5.2 

Cost of Scholarships Paid and Offered, by Scholarship Type, Over the Life of 
the Scholarship 

101 

5.3 

Cost of Scholarships Paid and Offered, by Payment Type, Over the Life of the 
Scholarship 

103 

A.1 

CONSORT Statement Flow Diagram 

122 

B.l 

Sample CFC-PBS Scholarship Award Letter 

135 


viii 



B.2 Sample CFC-PBS Control Group Notification Letter 138 

Box 

1 . 1 Cash for College Partnerships 1 1 

2.1 How to Read the Impact Tables in This Report 34 

4.1 How to Read the Six-Way Impact Tables in This Report 70 

5. 1 Scholarship Types Analyzed in Chapter 5 94 

5.2 Marginal Cost Methodology 97 

5.3 Factors Influencing the Administrative Cost of Performance-Based 

Scholarship Programs 100 


IX 




Preface 


From the White House to the schoolhouse, policymakers, educators, and citizens alike are 
calling for more college graduates across the country. President Obama has set the lofty goal of 
graduating a higher proportion of citizens than any other nation in the world by 2020. Reaching 
this level is a daunting task: Only 59 percent of students entering four-year institutions graduate 
within six years, and just 31 percent of students entering two-year institutions graduate within 
four years. The completion rate for low-income students is even bleaker, as they face barriers 
such as limited resources to manage the various costs associated with college, competing 
demands on their time (such as work or child-care responsibilities), and insufficient preparation. 
One of the ways that this administration has tackled these obstacles is to expand federal support 
— primarily Pell Grants to help more students afford college. 

State and private donors also contribute more than $16 billion in scholarships, many of 
which use merit-based criteria that reward high school performance. What do we know about 
the most effective way to structure these scholarship programs? Little research has been done to 
understand how creative financial aid structures might better assist students, especially low- 
income, disadvantaged students who might not meet merit-based criteria. With support from a 
consortium of foundations, colleges, and intermediaries, MDRC has worked in six states with 
over 12,000 students to test several different scholarship designs. All of these subsidies are 
known as performance-based scholarships — need-based grants, contingent on meeting 
academic benchmarks in college, that are intended to reduce the cost of college for low-income 
students and create incentives for academic success once they are enrolled. 

This report presents early findings from an evaluation of an innovative application of 
performance-based scholarships that targeted college-bound high school seniors in California, 
referred to as the Cash for College Perfonnance-Based Scholarship Program. The program 
recruited over 5,000 students into the study and randomly assigned them to be eligible for one 
of six different scholarships (ranging from $1,000 to $4,000 and from one term to two years) or 
a control group. The scholarships were completely portable, meaning that a student could use 
them at any accredited, degree-granting college or university. The findings indicate that such a 
statewide program could be implemented well, enabling more students to receive some funding 
toward their college costs. Additionally, the program encouraged students to matriculate, mainly 
at two-year institutions. While there is evidence that the program had positive impacts on 
academic success, it did not boost persistence in college after the students’ initial registration. 
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More time is needed to investigate whether these early results will translate into impacts 
on graduation and to understand whether certain scholarship amounts and durations work better 
for students. However, these early results and the design of this study lay an important founda- 
tion for understanding how we can better configure existing scholarship programs. 

Gordon L. Berlin 
President, MDRC 
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Executive Summary 


Students, families, educators, and policymakers alike realize the growing importance of a post- 
secondary degree or credential in competing for jobs. In fact, many state economic projections 
show a steadily increasing demand for a highly educated workforce. Yet policymakers and edu- 
cation leaders are grappling with how to boost college attendance and completion in an era 
when the number of college graduates is lagging behind demand and government resources are 
increasingly limited. 1 Increasing the number of college graduates is particularly difficult given 
the continued rising cost of attending college and the failure of financial aid to keep pace, which 
comiects to both access and retention. The combination of these factors particularly affects low- 
income students, who have limited means to pay for college. 

While the Pell Grant is the main federal source of need-based aid, states and private do- 
nors together contribute more than $16.4 billion in scholarships to undergraduates. 2 Many of 
these scholarships are structured so that they go to students who already have a high chance of 
success. In contrast, performance-based scholarships are need-based grants with payment con- 
tingent on meeting certain academic benchmarks in college. These types of scholarships have 
the potential to help students who might not otherwise qualify for merit aid, which is often giv- 
en based on high school performance. With anchor funding from the Bill & Melinda Gates 
Foundation and a consortium of other foundations, MDRC has worked in six states with over 
12,000 students, eight institutions, and one intennediary to test several different performance- 
based scholarship designs and to address on a much larger scale and in a wide range of settings 
the question of whether this innovative form of financial aid can improve academic achieve- 
ment in both the short term and long term. 

This report presents early findings from an ambitious evaluation of perfomiance-based 
scholarships targeting college-bound high school seniors across the state of California, referred 
to as the Cash for College Performance-Based Scholarship (CFC-PBS) Program. Building on 
an existing program geared to induce students to complete their Free Application for Federal 
Student Aid (FAFSA), CFC-PBS recruited over 5,160 students to participate in the chance to 
earn scholarships that ranged from $1,000 to $4,000. Students were eligible for one of six dif- 
ferent scholarship types over a period from one semester to four semesters, so that researchers 
could leam what amounts and durations of scholarships work best for students. The scholar- 
ships were completely portable — not tied to attendance at a particular institution — and could 


'Organisation for Economic Co-operation and Development, Education at a Glance 2007: OECD Indica- 
tors (Paris: OECD Publishing, 2007); College Board, Trends in Student Aid 2012 (New York: The College 
Board, 2012). 

2 College Board (2012). 
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be taken to any accredited, degree-granting college or university in the country. The program is 
evaluated using a random assigmnent design, similar to that used in medical efficacy trials. The 
specialized nature of the program and rigor of the evaluation design provide unique causal in- 
formation on the role of financial aid in matriculation. 

This report analyzes three terms of follow-up data from the program in California to 
address these research questions: 

• Is it feasible to implement a statewide, portable scholarship program that 
makes aid contingent on students’ perfonnance in college? 

• Do students who are offered performance-based scholarships have better ac- 
ademic outcomes than similar students offered scholarships without perfor- 
mance conditions attached? 

• What does it cost to implement a performance-based scholarship program? 

This report finds that the CFC-PBS Program was largely implemented as designed, en- 
couraged more students to matriculate, and had positive impacts on academic success. 


The CFC-PBS Program 

The program in California is part of a multistate evaluation of perfonnance-based scholarships 
taking place at institutions in Arizona, Florida, New Mexico, New York, and Ohio. The Per- 
formance-Based Scholarship (PBS) Demonstration is testing different versions of these scholar- 
ships, and each state represents a program with a different target population, scholarship design, 
and perfonnance benchmark. However, all programs share a few basic tenets: Payments are 
based on registering for a certain number of credits and earning a “C” or better at the end of the 
tenn; the scholarships are paid directly to students rather than to their institutions; and the schol- 
arship dollars are paid on top of the federal Pell Grant, state-based aid, and any other aid for 
which students are eligible. 

The program in California was built into an existing statewide program called Cash for 
College, a public -private partnership effort co-led by the California Student Aid Commission 
and the Los Angeles Area Chamber of Commerce and its affiliate, UNITE-LA, with funding 
provided by the College Futures Foundation. The Cash for College program brings together 
high schools, colleges, communities, businesses, and local government organizations and agen- 
cies to help low-income youth successfully complete the college financial aid application pro- 
cess (consisting of completing the FAFSA and the application for the California state financial 
aid program, Cal Grant), with the goal of helping to maximize the amount of state and federal 
aid that students can obtain. Between December and March of each academic year, college fi- 
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nancial aid staff members, high school counselors, and trained community volunteers assist stu- 
dents and families at hundreds of Cash for College workshops throughout the state. 


Prior to the evaluation of perfonnance-based scholarships, Cash for College workshops 
offered a $1,000 scholarship (not perfonnance-based) to a randomly selected attendee at every 
workshop who completed a financial aid application as an incentive to attend. MDRC collabo- 
rated closely with the Cash for College partners and the College Futures Foundation to develop 
an ambitious plan to test not only this Cash for College scholarship but an entire suite of per- 
fonnance-based scholarships, with the goal of detennining which configuration would produce 
the greatest impact on academic outcomes. During the study period (from spring 2009 to sum- 
mer 2012), students were randomly assigned to one of several different scholarship groups that 
varied in key features: 


Scholarship Type 1: Original Cash for College scholarship of $1,000 over 
one tenn with no perfonnance incentive, described above. 3 

Scholarship Type 2: Perfonnance-based scholarship of $1,000 over one 
tenn. 

Scholarship Type 3: Perfonnance-based scholarship of $1,000 over one 
year. 

Scholarship Type 4: Perfonnance-based scholarship of $2,000 over one 
year. 

Scholarship Type 5: Perfonnance-based scholarship of $2,000 over two 
years. 

Scholarship Type 6: Perfonnance-based scholarship of $4,000 over two 
years. 


The evaluation also randomly assigned some students to a control group that was not el- 
igible for a scholarship. Random assignment ensured that students in all the groups had similar 
levels of academic preparation, motivation, and other characteristics at the start of the study. By 
tracking students in the different groups over time and comparing their outcomes, researchers 
can determine the “value added,” or impact, of different types of scholarships on college en- 
rollment, credits attempted and earned, graduation, and other outcomes. As a result of the vari- 
ous durations of the scholarship types, the three semesters of follow-up analyzed in this report 


’Students who were eligible for this scholarship type received the funds at the start of the semester, so they 
were not subject to the satisfactory academic progress (SAP) requirements of their institutions, which often 
require the maintenance of a certain grade point average (GPA) and attainment of a certain pass rate for courses 
attempted. 
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show effects that were observed for some scholarships after the program period had ended and 
for others that had not yet reached the end of the intervention. 


The CFC-PBS Study Sample 

MDRC and Cash for College recruited students for the evaluation over a two-year period. To be 
eligible, students had to be high school seniors, attend a Cash for College workshop in a geo- 
graphic region targeted for the study, complete the federal and state financial aid application 
process, and meet certain income thresholds. 4 Intake began in 2009 in Los Angeles County and 
a vast area referred to as the Far North region, encompassing 1 1 rural counties below the Ore- 
gon border. In 2010, the evaluation expanded to include the Capital region around Sacramento 
and Kem County in California’s agricultural Central Valley. 

The analysis in this report focuses on over 5,000 students who met the eligibility criteria 
and were randomly assigned to one of six program groups or a control group. Students who at- 
tended workshops in the Los Angeles region make up over half of the sample, while around 
one-fourth of the sample are Far North workshop attendees. Over three-fifths of sample mem- 
bers identified themselves as Hispanic/Latino, and around one-fifth of sample members identi- 
fied as white. Over half of the CFC-PBS sample reported that they are the first member of their 
family to attend college. 

Baseline financial aid data were also collected for all students in the sample and showed 
that around half received some public benefits (mostly free or reduced-price lunches or food 
stamps). 5 Most students had financial circumstances that qualified them for the federal Pell 
Grant. Around 65 percent of students were eligible for a Cal Grant award. Together, the finan- 
cial aid and Cal Grant characteristics indicate that the sample is made up primarily of low- 
income, traditionally aged, dependent students with high levels of financial need. 

The CFC-PBS Program sample was not designed to represent all California first-year 
students. It was drawn out of a population of students who actively chose to attend a Cash for 
College workshop and opted to be considered for the study. This self-selecting group of fairly 
motivated students was drawn while students were still in high school applying for admission 
to college. 


4 Students had to be below the Cal Grant A and C income thresholds defined by the state of California. 

5 The former Food Stamp Program is now the Supplemental Nutrition Assistance Program (SNAP), but 
SNAP benefits are often referred to as “food stamps,” and the two terms are used interchangeably in this report. 
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Program Implementation 

The program design for the CFC-PBS study was quite complex. During the 2010 intake period, 
10,624 students participated in 210 Cash for College workshops in the targeted regions and 
9,765 participants were interested in being part of the study. (A total of 15,420 students ex- 
pressed interest in participating in the study over the two-year intake period.) In addition, eligi- 
ble participants were randomly assigned to one of six program groups or a control group, and 
the various verification and disbursement processes were modified to address the dramatic in- 
crease in the volume of scholarship disbursements (almost three times larger than the volume 
given out in 2008-2009). 

• The CFC-PBS Program was largely implemented as designed. 

The existing structure of Cash for College provided a strong foundation for the CFC- 
PBS Program. The scholarship program complemented the Cash for College program, attesting 
that a large-scale portable scholarship program can be implemented successfully. The study re- 
quired additional procedures in order to manage the various scholarship types, the increased 
volume of program group students, and the additional requirements attached to the disburse- 
ment of the award. 

• On average, across all six scholarship types, students received about 
$900 in scholarships over the first year and the initial enrollment pay- 
ment of the second year. 

Scholarship disbursements were divided into two categories: those made upon proof of 
a student enrolling in a college or university, and those made upon proof of a student meeting 
the performance requirements of a “C” or better GPA in at least six credits. About 83 percent of 
program group members received at least one scholarship payment over the program’s first 
three terms. On average, program group members received about $900 in scholarships, which 
represents slightly more than 63 percent of the total scholarship amount for which they were 
eligible. Because of the design of the scholarship, students had to submit documentation to 
show that they had enrolled at the beginning of a term and then that they had achieved the re- 
quirements to receive the award; therefore, there may be a subset of students who met the 
scholarship benchmark but did not submit their documentation, and thus did not receive their 
payment. Figure ES.l outlines the average scholarship amount received by program group 
members for each scholarship type. 

Students who received CFC-PBS payments reported that the money was primarily used 
toward college-related expenses — books being the most commonly purchased item. They also 
mentioned that the timing of the disbursements of CFC-PBS moneys helped cover up-front 
costs that were incurred before financial aid was disbursed. 
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Scholarship amount ($) 


Figure ES.l 

Average Scholarship Amount Received by Program Group Members: 
First Through Third Terms 

Cash for College Performance-Based Scholarship Study 
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Figure ES.l (continued) 


SOURCES: MDRC calculations using scholarship payment data provided by the California Student 
Aid Commission as well as data from the U.S. Department of Education, National Center for 
Education Statistics, Integrated Postsecondary Education Data System (1PEDS). 

NOTES: The figure does not include performance payments in the third term, as fall 2010 cohort data 
were not available at the time of data acquisition. 

Dotted lines above the bars represent the maximum amount of money that students were eligible 
for over the first two terms and the enrollment payment of the third term. 

The percentage of the maximum possible award earned over the first three terms is reported in 
parentheses under the average amount received over the first three terms. 

Students assigned to Scholarship Type 1 are eligible for an enrollment scholarship payment only in 
the first term. 

Students assigned to Scholarship Type 2 are eligible for scholarship payments only in the first 
term. 

Students assigned to Scholarship Types 3 and 4 are eligible for scholarship payments only in the 
first and second terms. 


• The scholarships’ benchmark of a GPA of “C” or better seemed attain- 
able to large numbers of students. 

Field research was conducted to learn about students’ experiences in college and their 
perceptions of the CFC-PBS Program. The majority of focus group participants mentioned that 
while the extra money was beneficial, they did not feel the scholarship by itself made a signifi- 
cant difference in motivating them academically. However, knowing that they could lose the 
award did appear to give a considerable number of students an incentive to work harder, based 
on interviews and survey results. Survey respondents were fairly split between those who were 
encouraged by the scholarship to take more classes and those who were unaffected by it. 


Findings on Academic Outcomes 

The academic findings are presented for students’ first year in college, as well as the first term 
of their second year. Sample members were surveyed about 12 months after they entered the 
study to provide information about their levels of effort in their studies, their motivation levels, 
and their employment patterns. Overall, the program generated some modest but positive early 
effects on student outcomes; a future report will provide effects on longer-term outcomes, such 
as graduation. Moreover, since this report does not present data for all of the tenns during 
which students were eligible for an award, it is premature to draw final conclusions about the 
effectiveness of these awards or to determine what amounts of aid matter based on the analyzed 
data. Specifically, the early findings show the following: 
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• The CFC-PBS Program encouraged more students to matriculate. 

The CFC-PBS Program is one of the first studies to document a causal relationship be- 
tween financial aid and matriculation. Although the program was marketed mainly to students 
who had already taken steps to enroll in college (namely, attending a Cash for College work- 
shop for completing financial aid applications), the scholarship offer resulted in about a 6 per- 
cent increase in matriculation among program group students (by 4.7 percentage points over the 
control group mean of 84.4 percent). Figure ES.2 shows that the matriculation impacts are con- 
centrated among students attending two-year colleges (specifically, California community col- 
leges). This is reasonable given the timing of the notification of scholarship eligibility: Letters 
were sent out in June, in the summer prior to fall matriculation but after the deadlines for ac- 
ceptance to most four-year institutions. Because community colleges have flexible start dates, 
this suggests that the program induced students who were not going to attend school to change 
their decision and enroll in a community college. 

These effects extend to numerous subgroups, such as males, females, and students with 
high school GPAs below and above 3.0. There is compelling evidence that the intervention had 
larger effects on enrollment for those students with lower high school GPAs than for students 
with higher high school GPAs. Lastly, the program had stronger effects for students who may 
not have been intrinsically motivated to apply for financial aid. (In other words, the program 
seemed to have a greater effect on those students who attended the workshops based on parental 
pressure, counseling pressure, or other external pressures.) 

• Positive impacts of the program seem to be concentrated among stu- 
dents attending two-year colleges (specifically, California community 
colleges). 

In addition to increasing matriculation at community colleges, the program also mod- 
estly increased persistence at such colleges. During the second term after random assigmnent, 
students in the program group were 3.7 percentage points more likely to register compared with 
the control group mean of 84.7 percent. The small increases in enrollment continued into the 
third term as well. 

The program also induced students attending California community colleges to attempt 
and to earn a greater number of college-level credits. However, the credits attempted and earned 
are relatively small in magnitude (about one-quarter of a course). 

• The program had positive impacts on students’ participation in activi- 
ties aimed at helping them improve their academic performance. 

Students who were eligible for scholarships were more likely to take courses on im- 
proving their study skills, seek academic services outside of class, and participate in study 
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Figure ES.2 (continued) 


SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

National Student Clearinghouse data were not found for 320 students (6.5 percent of the 
sample). 


groups. The program also seems to increase the “good” types of extrinsic motivation: The sur- 
vey findings suggest that the money from the intervention likely motivates students to achieve 
academically, and the motivation may remain when the stimulus of the money is removed. 
Lastly, there is some evidence that the intervention decreased employment, giving students 
more time to focus on their studies. 


Program Cost 

Cost estimates are based on actual scholarship payments and program expenditures over two 
years (from January 2009 to June 2011). The cost analysis estimates the cost of Scholarship 
Types 1 through 4 for the period covered in this report. Cost analyses over the full period of the 
scholarship will be published in a later report. The analysis describes costs that are required to 
continue operating the program, as it is believed that this provides the most informative estimate 
for policymakers, foundations, colleges, or others who may be interested in implementing such 
a program. As a result, all costs related to MDRC’s evaluation have been excluded, as have 
start-up costs. 6 Overall, the analysis suggests the following: 

• The conditional nature of performance-based scholarships results in the cost 
of scholarship payments being noticeably lower than the dollar amount of the 
scholarships offered to students. The difference between the amount offered 
and the amount paid widened as scholarship performance requirements were 
spread across longer periods of time (hence, with a larger number of total 
perfonnance criteria). That said, there may have been some students who met 
enrollment or perfonnance benchmarks but did not receive a payment be- 
cause they did not submit the necessary verification. The size of this group of 
students may have increased over time. 


6 For example, this analysis excludes costs related to the design and development of the PBS Scholars 
website blit includes costs related to its ongoing maintenance over the analysis period. Start-up costs can be 
expected to vary widely from setting to setting depending on the existing infrastructure and resources available. 
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• The cost to administer scholarships increased as performance requirements 
were added, but the decrease in payments to students more than offset the in- 
creased cost of administration. All else being equal, scholarships with more 
performance requirements cost less than scholarships with fewer perfor- 
mance requirements. 

Placing These Findings in Context 

A number of factors may contribute to the results summarized above. First, the budget climate 
in California during the program operation period was marred by funding cuts, which may have 
created a sense of urgency among students and their parents. Second, the timing of the program 
— randomly assigning students to scholarship groups while they were still in their senior year 
of high school, just before graduation — may also have been a strong contributing factor, 
prompting students on the verge of nonattendance to change their minds. Finally, the results 
may reflect the increase in the financial aid given to students, similar to other sites in the PBS 
Demonstration. 

Given that the program operated very much like an external scholarship program (that 
is, one not linked to a particular institution), it is interesting to see how the findings compare 
with others in the literature. Overall, the magnitude of the CFC-PBS findings is in alignment 
with those reported previously in other studies, indicating that relatively small scholarship 
amounts could be effective in increasing matriculation among students if the scholarships are 
well designed and targeted effectively . 7 Interestingly, the CFC-PBS Program also compares 
favorably with the other sites in the PBS Demonstration. This is notable as it may have had the 
lightest interaction with students as a result of the scholarship being completely portable and 
implemented by a statewide partnership, while other programs in the PBS Demonstration have 
been based at institutions. Notably, all of the Demonstration sites, with the exception of New 
Mexico, have been focused on community colleges, and the positive results from the study in 
California have been driven almost entirely by students attending community college. 


7 Susan Dynarski, “Does Aid Matter? Measuring the Effect of Student Aid on College Attendance and 
Completion,” American Economic Review 93, 1 (2003): 279-288; Thomas J. Kane, “A Quasi-Experimental 
Estimate of the Impact of Financial Aid on College-Going,” NBER Working Paper 9,703 (Cambridge, MA: 
National Bureau of Economic Research, 2003). 
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In general, impacts for performance-based scholarship programs are slightly more posi- 
tive than other incentive-based programs found in the literature. 8 This may reflect the targeting 
of programs; on average, students in the PBS Demonstration have one or more risk factors for 
not completing college, such as being low-income, older, parents, and so on, which may con- 
tribute to the larger effect of the contingent grant on academic outcomes. Finally, the analysis 
indicates that the CFC-PBS Program impacts in particular are sizable relative to those reported 
in other studies. This suggests that larger impacts from these types of designs may not result 
from larger infusions of money, but rather from better targeting of students who may be respon- 
sive to incentives and consideration of the components of the incentive program itself. 


Policy Implications and Conclusions 

The California Cash for College program provided distinctive features conducive to a large- 
scale experimental study: (1) the program was able to reach a large group of students, (2) the 
workshops were held in communities that are low-income or have low college-going rates to 
ensure that scholarships targeted students at the highest risk of not matriculating, and (3) the 
original practice of offering a $1,000 scholarship to one attendee at every workshop provided a 
platform for adding other scholarships that could be awarded at random. While the strong Cash 
for College partnership may be distinctive, similar approaches may be possible for state agen- 
cies or large, private scholarship providers. In this way, the results speak to both the ability to 
implement such a program on a large scale and the efficacy of the strategy in helping students 
persist and be academically successful in college. 

Implications for National Policy 

Since perfonnance-based scholarships seem to improve outcomes for students, some 
policymakers might ask whether it would be a good idea to tie federal financial aid payments 
such as the Pell Grant more closely to achievement. 

In some ways, the Pell Grant program is already tied to perfomiance: Students remain 
eligible for their Pell Grants by meeting satisfactory academic progress (SAP) requirements. 
The exact SAP criteria vary by institution, but in most cases this means maintaining a GPA of at 
least 2.0. But there are important differences between this standard and those used in perfor- 
mance-based scholarships. With Pell Grants, performance consequences come with a time lag: 
Students who fail to meet SAP requirements may see their Pell eligibility revoked the following 
semester or academic year. In contrast, perfonnance-based scholarships reward performance 

s Richburg-Hayes, L., “tncentivizing Success: Lessons from Experimenting with Incentive-Based Grants,” 
pages 101-126 in Andrew Kelly and Sara Goldrick-Rab (eds.), Reinventing Financial Aid (Cambridge, MA: 
Harvard Education Press, 2014). 
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immediately at the end of the semester: If students do not meet their benchmarks in a given 
tenn, they do not receive a payment. 

There are a few reasons to be cautious about drawing conclusions for Pell Grants from 
the results of performance-based scholarships, however. First, the Pell Grant is far more gener- 
ous than the perfomiance-based scholarships studied here, and as such, it is generally the foun- 
dation of a student’s financial aid package. Performance-based scholarships are paid in addition 
to all other financial aid. For example, both the program group and the control group in the 
CFC-PBS project received significant base levels of financial aid. This means that changing 
disbursement criteria for Pell Grants could change students’ behavior in quite different ways 
from the CFC-PBS project. 

Second, structuring Pell Grants more like performance-based scholarships could have a 
chilling effect on enrollment. The Pell Grant is generally paid all at once, near the beginning of 
the semester. Students often use it to pay for tuition and fees first, before other educational ex- 
penses. Performance-based scholarships, on the other hand, are paid in increments, and could 
help with other educational expenses, such as room and board, books, and transportation over 
the semester. These expenses are important, but only after the primary expense of tuition is cov- 
ered at the outset of the semester. Therefore, an incremental payment model could bar many 
low-income students from the higher education system altogether and create a lot of upheaval 
for colleges, which are unlikely to have other funds to assist students who need a stable source 
of aid. 


The question of whether the current Pell program should be more performance-based is 
another study entirely. The PBS Demonstration was designed to answer the question of whether 
additional financial aid, made contingent on certain academic benchmarks, could help low- 
income students progress through college. It was not a test of whether a large, existing, need- 
based grant program, such as the Pell Grant program, should be similarly perfonnance-based. 
Additional research is needed before infonned decisions could be made about such a large 
change . 9 


9 A current evaluation of Aid Like A Paycheck — an intervention in which financial aid refunds are dis- 
bursed every two weeks during the semester — is testing the effects of changing the timing of Pell Grant dis- 
bursements to be more closely aligned with when aid is earned. By providing students with their refunds even- 
ly throughout the term, the program may help students better manage their time and money. As with perfor- 
mance-based scholarships, the incremental disbursements of aid may also give students an incentive to contin- 
ue in their classes in order to receive all of their financial aid. See Michelle Ware, Evan Weissman, and Drew 
McDermott, Aid Like a Paycheck: Incremental Aid to Promote Student Success (New York: MDRC, 2013). 
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Implications for State Policy in California 

Budgetary shortfalls and the recent recession spanning 2007 to 2009 have hurt Califor- 
nia’s community colleges and universities. The demand for postsecondary education has in- 
creased at the same time that the state’s support has been reduced. Although the passage of Cali- 
fornia Proposition 30 during the 2012 election cycle helped to stabilize the financial situation in 
the state, public education funding has only returned to levels comparable to those before the 
economic collapse. 10 

In such a budgetary situation, accompanied by increased demand for financial aid, sys- 
tematically studying the effect of randomly implementing performance-based aid or exploring 
alternative distribution patterns may be prudent policy. For example, policymakers may be 
able to experiment with perfonnance criteria by lowering the high school GPA threshold and 
permitting students who would not otherwise be eligible to earn a perfonnance-based Cal 
Grant. 1 1 This could potentially increase the proportion of low-income students who matricu- 
late. Alternatively, instead of raising the Cal Grant high school GPA threshold (as was pro- 
posed during the 2012-2013 budget discussions), the original GPA level could be maintained if 
the scholarship were perfonnance-based. 12 In this way, more students would be served despite 
restrictions to eligibility. 

Implications for Scholarship Providers 

Private scholarship providers often give scholarships based on various eligibility criteria 
that may or may not include financial need. Indeed, some of these scholarships are administered 
somewhat haphazardly. 13 They often do not have a specific goal or, as with merit-based scholar- 
ships, they go to students who already have a high chance of academic success. 

Scholarship providers that are able to experiment with perfomiance-based aid programs 
could help answer questions about how perfonnance-based scholarships affect students and 
how these scholarships might help the providers accomplish their goals. Perfonnance-based 
scholarships could also help traditional scholarship programs maximize the amount of money 


10 Jessica Calefati and Josh Richman, “Proposition 30: A Year Later, California Schools Seeing Benefits of 
Tax Measure,” San Jose Mercury News (November 2, 2013). 

"While there are some performance criteria with the existing Cal Grant program, they vary and their ap- 
plication can be delayed based on institutional requirements. Performance-based programs in this context im- 
ply a greater frequency and earlier checks on performance adherence. 

12 Mac Taylor, The 2012-13 Budget: Analysis of the Governor’s Higher Education Proposal (Sacramento, 
CA: Legislative Analyst’s Office, 2012). 

13 Jennie H. Woo and Susan P. Choy, Merit Aid for Undergraduates: Trends from 1995-96 to 2007-08, 
NCES 2012-160 (Washington, DC: U.S. Department of Education, National Center for Education Statistics, 
2011 ). 
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they are able to offer, because students are offered the opportunity to earn more scholarship dol- 
lars than they actually do earn on average. But here, too, there could be trade-offs if dependable, 
non-perfonnance-based money often makes it possible for students to enroll who otherwise 
would not. 


Next Steps for the Project 

A future report will present a cross-site synthesis of the final results from this and other sites 
from the PBS Demonstration programs. Importantly, it will provide more follow-up on the 
longer scholarship types to provide more insight into how scholarship amounts and duration can 
influence student outcomes. 
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Chapter 1 

Introduction 


Policymakers and education leaders are grappling with how to boost college attendance and 
completion in an era when government resources are increasingly limited. In California, the 
concern is driven in part by projections of the state’s economy that show a steadily increasing 
demand for a highly educated workforce, and by a realization that the state is lagging behind in 
the number of students who graduate from college. If current trends persist, in 2025, 41 percent 
of California jobs will require at least a bachelor’s degree, while only 35 percent of working-age 
adults in the state will have one. 1 Arguing that the state needs the innovation and productivity 
that comes from a well-educated workforce, a nonpartisan group of civic and business leaders 
issued a call for California to award 2 million more postsecondary credentials by 2025. The 
group singled out three main strategies to achieve that goal: improving high school completions 
and the transition to college, closing the racial and ethnic achievement gap in college, and im- 
proving student success in California’s community college system. 2 

This report presents preliminary findings from an ambitious evaluation designed to test 
the effectiveness of different configurations of scholarships in improving academic outcomes 
for college-bound high school seniors in California. In contrast to merit scholarships offered 
only to the top students, the scholarships in this study were awarded to students with a wide 
range of academic backgrounds and abilities and could be used at any accredited college or uni- 
versity that students chose. Most of the scholarships were performance-based, meaning that stu- 
dents had to demonstrate that they could maintain at least a half-time course load and a “C” 
grade point average (GPA) in order to receive full funding. The report analyzes three terms of 
follow-up data to address these research questions: 3 

• Is it feasible to implement a statewide, portable scholarship program that 
makes aid contingent on students’ perfomrance in college? 

• Do students who are offered performance-based scholarships have better ac- 
ademic outcomes than similar students offered scholarships without perfor- 
mance conditions attached? 


'Johnson (2014). 

California Competes (2012). 

3 Given the various durations of the scholarship types, the follow-up period results show effects observed 
after the program ended for some scholarships and ongoing effects for other scholarships that had not yet 
reached the end of the intervention. 
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What does it cost to implement a perfonnance-based scholarship program? 


To answer these questions, the evaluation used a process in which students were ran- 
domly assigned to one of several different scholarship groups that varied in key features. For ex- 
ample, all but one of the scholarships had perfonnance criteria attached. The amount of money 
offered ranged from a total of $1,000 to $4,000, and the scholarships lasted from one academic 
tenn to two academic years. The evaluation also randomly assigned some students to a control 
group that was not eligible for a scholarship. Random assignment ensured that students in all the 
groups had similar levels of academic preparation, motivation, and other characteristics at the 
start of the study. By tracking students in the different groups over time and comparing their out- 
comes, researchers can determine the “value added,” or impact, of different types of scholarships 
on college enrollment, credits attempted and earned, graduation, and other outcomes. 

The California Cash for College Perfonnance-Based Scholarship (CFC-PBS) study de- 
scribed in this report is part of a larger, multistate initiative called the Perfonnance-Based 
Scholarship (PBS) Demonstration. While slightly different programs were implemented and 
evaluated in Arizona, Florida, New Mexico, New York, and Ohio, all programs share common 
features: providing an opportunity for targeted students to receive perfonnance-based aid, pay- 
ing earned scholarships directly to students (as opposed to their institutions, to be applied to 
their student accounts), and supplementing the existing financial aid for which students are eli- 
gible (resulting in a net increase in financial aid). 

MDRC, a nonprofit education and social policy research organization, is managing the 
demonstration and conducting the evaluation, with major funding provided by the Bill & 
Melinda Gates Foundation. A consortium of philanthropies, government agencies, postsecondary 
institutions, and nonprofit organizations is providing financial and programmatic support. In Cal- 
ifornia, the lead partners include the College Futures Foundation, which provides the funds for 
the scholarships; Cash for College, a statewide program of financial aid workshops that provided 
a platfonn for recruiting students into the study; and the Los Angeles Area Chamber of Com- 
merce/UNITE-LA, which administered the perfonnance-based scholarships. 

The remainder of this chapter is organized as follows: It begins with a brief overview of 
the PBS Demonstration and explains the history and structure of the CFC-PBS Program (the 
California portion of the demonstration). This is followed by a discussion of the evaluation de- 
sign. Next, the chapter provides some context for the evaluation by explaining the higher educa- 
tion landscape in California and the difficult economic conditions under which the evaluation is 
taking place. The chapter concludes by walking through the program’s theory of change and 
laying out the organization of the rest of the report. 
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A Brief Overview of the PBS Demonstration 

The PBS Demonstration was conceived and funded in response to growing national concern 
about college access and completion. While college enrollment in the United States has grown 
steadily over the past several decades, there are persistent gaps among high school graduates 
from different income levels. In 2009, only 55 percent of high school graduates from families at 
the bottom fifth of the income distribution went to college; among middle-income and the most 
affluent families, the rates were 67 percent and 84 percent, respectively. 4 Once they are enrolled 
at a college or university, a distressingly high percentage of degree-seeking students make slow 
progress or drop out. Research by the U.S. Department of Education finds that only 35 percent 
of incoming students at community colleges — and 65 percent of incoming students at four- 
year institutions — earn a certificate or degree from any college or university after six years. 5 

There are many reasons why more Americans do not complete college, but a major fac- 
tor is cost. Over the past decade, tuition and fees at public colleges and universities across the 
United States have risen much faster than the rate of inflation. For example, between the 2001- 
2002 and 201 1-2012 academic years, published tuition and fees for community college students 
increased at an average rate of 3.8 percent a year beyond the rate of general inflation; for in- 
state students at public four-year institutions, the annual increase was even higher (5.6 percent). 6 
While many students and their families pay less than the published prices because of financial 
aid, the “sticker price” can have a negative effect on college attendance desires. Moreover, the 
cost of living while going to school has also continued to rise, and grant aid is rarely sufficient 
to meet those costs. 7 Some students decide that college is not worth the expense; others are 
forced to borrow money or increase the hours they work while in school, possibly to the detri- 
ment of their academic performance and completion of degrees. 

Performance-based scholarships offer a strategy to supplement existing financial aid 
programs in a way that may help make college more affordable for students and give them an 
incentive to make greater progress. The idea was first developed by MDRC and the state of 
Louisiana under the auspices of the Opening Doors Demonstration, a project launched by 
MDRC in 2004-2005 to identify strategies to increase persistence and academic success among 
students in community college. 8 Low-income parents at two community colleges in the New 
Orleans area were offered $1,000 for each of two semesters ($2,000 total) on the condition that 


4 Baum, Kurose, and McPherson (2013). 

5 Radford, Berkner, Wheeless, and Shepherd (2010). 

6 College Board (2011). 

’College Board (2013). 

8 Brock and LeBlanc (2005). 
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they enrolled at least half-time and maintained a “C” or better GPA. Similar to the PBS Demon- 
stration, MDRC conducted a random assignment evaluation to measure the impacts of the pro- 
gram, and found that students who were offered the scholarship had significantly higher rates of 
enrollment and attempted and earned more college credits than students in a control group. The 
evaluation also found that the scholarships had positive social and psychological effects, includ- 
ing helping students feel more positive about themselves and their futures. 9 

The Louisiana study ended prematurely in 2005 because of the devastating impact of 
Hurricane Katrina, but the positive results from the program led MDRC to launch the PBS 
Demonstration to find out whether the approach would work at other institutions and with oth- 
er types of students. The core idea remains the same: Scholarships are predicated on at least 
half-time enrollment and a “C” or better GPA, and are paid directly to students as a supple- 
ment to (not a substitute for) existing financial aid. At the same time, MDRC looked for states 
and institutions that were interested in testing variations of the Louisiana model. Some sites in 
the demonstration combine the scholarships with intensive advising or other services, or in- 
crease the financial reward if students attend full time rather than part time. Similarly, some 
sites target students straight out of high school, while others serve students who are older or 
who have developmental education requirements to meet. While most of the sites focus on 
two-year institutions, the study in California and New Mexico included students in four-year 
institutions. Table 1.1 shows the states where the PBS Demonstration is taking place and brief- 
ly describes the programs each state is testing. (The Louisiana Opening Doors program is also 
shown for comparison.) 

The CFC-PBS Program is arguably the most ambitious of all the sites in the demonstra- 
tion. As explained further in the next section, it offers six different types of scholarships in order 
to answer questions about their optimum amount and duration, and also to determine whether 
the addition of performance criteria (versus money alone) makes a difference in student out- 
comes. 10 California is also the only state in the demonstration that offers a portable scholarship, 
meaning that students can take the funds to any institution they choose — a two- or four-year 
institution, or even one outside of California. Finally, with over 5,000 students enrolled in the 
study, California has by far the largest sample of any site in the demonstration. 11 


9 Richburg-Hayes et al. (2009). 

10 While the data analyzed in this report are from a period that is too early to answer questions about the 
importance of amount and duration, the study was designed with this goal in mind. A future report will address 
the question of whether the amount or duration of scholarships (or both together) affect academic performance. 

1 'The analysis sample size in this report is 4,921, as discussed in Chapter 2. 


4 



The Performance-Based Scholarship Demonstration 
Table 1.1 


Summary of MDRC Performance-Based Scholarship Interventions 
Cash for College Performance-Based Scholarship Study 


Summary 

Opening Doors 



PBS Sites 



Characteristic 

Louisiana 

Arizona 

California 

Florida 

New Mexico 

New York 

Ohio 

Sample selection 
criteria 

Age 

18-34 

No age criteria 

16-19 

At least 18 

17-20 

22-35 

At least 18 

Low-income 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Additional 

requirements 

Must be a parent; 
earned a HS 
diploma, GED 
certificate, or passing 
score on college 
entrance exam 

Latino male; 
< 45 credits 
earned 

HS student; 
must have 
completed 
FAFSA and Cal 
Grant application 

In need of 
remedial 
math 

1 st-year 
student 

Live away from 
parents; 
in need of at least 
1 remedial course 

Must be a parent 

Intervention 

Maximum 
scholarship 
amount per term 

$1,000 

$1,500 

$333 -$1000 a 

$600 

$1,000 

$1,300 

$600 or $900 b 

Scholarship 

duration 

2 semesters 

3 semesters 

1 term to 2 years 

3 semesters 

4 semesters 

2 semesters and 1 
summer semester^ 

2 semesters or 3 
quarters 

Disbursements per 
term 

3 

3 

Varies 

2 

3 

3 

1 

Maximum amount 

$2,000 

$4,500 

$1,000 to $4,000 

$1,800 

$4,000 

$2,600 or $3,900 

$1,800 

Years of program 
operation 

Spring 2004 to 
Summer 2005 

Fall 2010 to 
Fall 2012 

Fall 2009 to 
Spring 2012 

Fall 2010 to 
Fall 2012 

Fall 2008 to 
Spring 2011 

Fall 2008 to 
Summer 2010 

Fall 2008 to 
Winter 2010 
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Table 1.1 (continued) 


Summary Opening Doors PBS 


Characteristic 

Louisiana 

Arizona 

California 

Florida 

New Mexico 

New York 

Ohio 

Performance 








benchmarks 








Academic criteria 

Enroll in college, and 

Part-time: 6- 

Enroll in 

Complete a 

Complete 12 

Enroll in 

Part-time: 6-11 


complete 6 or more 

11 credits 

college, and 

sequence of 

or more 

college, and 

credits with a 


credits with a “C” or 

with a “C” or 

complete 6 or 

math courses 

credits (1st 

complete 6 or 

“C” or better in 


better average 

better in each; 

more credits 

with a “C” 

semester) or 

more credits 

each; 



full-time: 12 

with a “C” 

or better in 

15 credits 

with a “C” or 

full-time: 12 



or more 

average or 

each course 

(subsequent 

better in each 

or more credits 



credits with a 

better 


semesters) 


with a “C” or 



“C” or better 



with a “C” 


better in each 



in each 



average or 








better 



Service criteria 

Meet with adviser 

Meet with 

None 

Complete 

Meet with 

None 

None 



adviser, 


tutoring 

adviser 





complete 


requirements 





tutoring and 
workshop 
requirements 
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Table 1.1 (continued) 



Opening Doors 




PBS 




Louisiana 

Arizona 

California 

Florida 

New Mexico 

New York 

Ohio 

Participating: 

institutions 

Delgado Community 
College 

Louisiana Technical 
College 

Pima 

Community 

College 

National Hillsborough 
(portable to Community 
accredited College 

institutions) 

University of 
New Mexico 

Hostos 

Community 

College 

Borough of 
Manhattan 
Community 
College 

Lorain 

County 

Community 

College 

Owens 

Community 

College 








Sinclair 

Community 

College 

Total sample 

537 d 

1,028 

4,921 e 

1,075 

1,081 

1,502 

2,285 


NOTES: Scholarships are limited to the institution where the evaluation occurred, with the exception of PBS California (the Cash for College 
Performance-Based Scholarship study). In that case, scholarships were portable to any accredited, degree-granting, two-year or four-year 
postsecondary institution in the United States. 

a The study in California randomly assigned program group members to one of six scholarship types that varied in amount (from $1,000 total to 
$4,000 total) and duration (from one term to two years). Students could take the award to any degree-granting, accredited institution in the 
country, and payments were adjusted to reflect the institution type (quarter or semester). 

b In the Ohio study the scholarship was worth up to $1,800 and the awards were divided evenly among two consecutive semesters or three 
consecutive quarters; thus, the maximum amount per term was $900 per semester or $600 per quarter. 

c The study in New York randomly assigned program group members to one of two scholarship types. One type was offered over two semesters 
only; the other was offered over two semesters plus one summer semester. 

d Although there were 1,019 study participants, only 537 participants from the first and second cohorts were analyzed, as Hurricane Katrina 
disrupted the follow-up period for the third and fourth cohorts. 

e Although there were 5,160 study participants, undocumented immigrant students were excluded from the analysis because of data reliability 
concerns. Thus, the analysis sample was 4,921 participants. 



History and Structure of the Cash for College Performance-Based 
Scholarship Program 

The CFC-PBS Program was built onto an existing statewide program called Cash for College, 
an initiative sponsored by the California Student Aid Commission (the agency responsible for 
administering the state’s financial aid programs) and run by a small paid staff and hundreds of 
volunteers. The goal of Cash for College is to inform high school seniors and their parents about 
financial aid opportunities and to help them complete the Free Application for Federal Student 
Aid (FAFSA) and the application for Cal Grant, the state’s major financial aid program. The 
underlying intent is to maximize the amount of state and federal aid for eligible students. In the 
winter and early spring of each year, Cash for College hosts hundreds of financial aid work- 
shops throughout the state that serve tens of thousands of students annually. The workshops are 
concentrated in schools and communities that are low-income or have low college-going rates 
(or both). In order to generate interest and boost attendance in the workshops, Cash for College 
offers a $1,000 scholarship (not performance-based) to at least one randomly selected attendee 
at every workshop who completes the FAFSA. 12 

MDRC first became aware of the Cash for College program in 2008, after being con- 
tacted by the College Futures Foundation (then called the College Access Foundation of Cali- 
fornia). At the time, the foundation was the major financial backer of Cash for College, and was 
interested in an evaluation to leam whether the $1,000 scholarship had any effect on student 
outcomes. After a few conversations, the foundation and MDRC developed an ambitious plan 
to test not only the Cash for College scholarship but an entire suite of performance-based schol- 
arships to determine which configuration would produce the greatest impact. The resulting pro- 
ject design led to a significant net increase in the total number of scholarships offered through 
Cash for College workshops during the 2009-2010 and 2010-2011 academic years and repre- 
sented a major advance in the overall research questions that could be answered by the PBS 
Demonstration. 

As noted earlier, the CFC-PBS study is using a random assignment design to test the ef- 
fectiveness of different types of scholarships. The scholarships include the following: 

• Scholarship Type 1: Original Cash for College scholarship of $1,000 over 
one term with no perfonnance incentive. 13 

• Scholarship Type 2: Performance-based scholarship of $1,000 over one 
term. 


12 See the Cash for College website (www.californiacashforcollege.org). 

13 Students eligible for this scholarship type received the funds at the start of the semester, so they were not 
subject to satisfactory academic progress requirements. 



Scholarship Type 3: Performance-based scholarship of $1,000 over one 
year. 


• Scholarship Type 4: Perfomiance-based scholarship of $2,000 over one 
year. 

• Scholarship Type 5: Perfomiance-based scholarship of $2,000 over two 
years. 

• Scholarship Type 6: Perfomiance-based scholarship of $4,000 over two 
years. 

In addition, some students were randomly assigned to a control group that was not of- 
fered any of the scholarships. Because the number of students who attended Cash for College 
workshops far exceeded the money available for scholarships, random assignment was the most 
equitable way to allocate the funds. Random assignment also provided the most reliable means 
for testing the impact of the scholarships, since it ensured that there were no systematic differ- 
ences between students assigned to the different groups at the start of the study. Random as- 
signment to one of the scholarship groups or to the control group did not affect students’ eligi- 
bility for federal Pell Grants, state Cal Grants, or other financial aid; therefore, the study exam- 
ines whether eligibility for the scholarship types listed above produces better outcomes for stu- 
dents than regular financial aid alone. 

MDRC and Cash for College recruited students for the evaluation over a two-year peri- 
od. To be eligible, students had to be high school seniors and attend a Cash for College work- 
shop in a geographic region targeted for the study. Intake began in 2009 in Los Angeles County 
and a vast area referred to as the Far North region, encompassing 1 1 rural counties below the 
Oregon border. In 2010, the evaluation expanded to include the Capital region around Sacra- 
mento and Kem County in California’s agricultural Central Valley. (See Figure 1.1 for the dis- 
tribution of Cash for College sites participating in the study.) Together, these four regions ap- 
proximated the rich diversity of the state. As described in Chapter 2, students had to complete 
the FAFSA by an early March deadline and demonstrate that they met the family income crite- 
ria and other guidelines set by the Cal Grant program in order to be randomly assigned. Over a 
two-year period, 15,420 students expressed interest in participating and 5,160 were randomly 
assigned to one of the scholarship groups or the control group. 14 

The program was implemented through strong partnerships. (See Box 1.1.) An affiliate 
of the Los Angeles Area Chamber of Commerce, UNITE-LA, collaborated with Cash for Col- 


14 The number of students who met the eligibility conditions exceeded the number required for evaluation 
purposes, so some students were randomly assigned to a nonstudy group. MDRC did not collect data on these 
students. See Chapter 2 for details. 
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The Performance-Based Scholarship Demonstration 

Figure 1.1 

Cash for College Workshop Locations 


Cash for College Performance-Based Scholarship Study 
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SOURCE: Performance-Based Scholarship Study Design in California. 
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Box 1.1 


Cash for College Partnerships 

Cash for College is a public-private partnership effort led by the California Student Aid 
Commission and the Los Angeles Area Chamber of Commerce and its affiliate, UNITE-LA. 
The partnership brings together high schools, colleges, communities, businesses, and local 
government organizations and agencies to help low-income young people successfully 
complete the college financial aid application process. Each year, college financial aid staff 
members, high school counselors, and trained community volunteers assist students and 
families at Cash for College workshops, held across the state. Key leaders already in place 
within Cash for College made it possible to implement the CFC-PBS Program. These lead- 
ers include the following: 

• A statewide coordinator at the California Student Aid Commission and cochair of a 
statewide advisory group, responsible for strategic planning, day-to-day operations, 
and coordination with MDRC 

• Regional coordinators, responsible for coordinating workshops, recruiting and training 
volunteers, and working with MDRC to reach the sample target 

• UNITE-LA and the Los Angeles Area Chamber of Commerce Foundation (the fiscal 
agent for Cash for College), responsible for verifying transcripts, disbursing and doc- 
umenting scholarship payments, communicating with students, and working closely 
with the statewide coordinator 

Cash for College has received support from the California Student Aid Commission, the 
College Futures Foundation, and the Kresge Foundation, and significant resources from re- 
gional and local partners. 


lege to administer all of the different scholarships . 15 Most students were notified of their re- 
search status in June of their senior year, shortly before graduation. If they were randomly se- 
lected to receive a scholarship, they were also given infonnation about the scholarship condi- 
tions and how to claim their award. To receive funds, scholarship recipients had to provide 
proof of enrollment at a college or university in the fall immediately after graduating from high 
school. For students in any of the perfonnance-based scholarship groups, the program also re- 
quired that they provide copies of their transcripts at the end of the academic tenn to show that 
they met the academic requirements. 


l5 Though it primarily focuses on promoting an effective educational system in Los Angeles, UNITE-LA 
performs a statewide role for Cash for College. More information on UNITE-LA is available on its website, 
www. unitela. com. 
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The performance-based scholarships were delivered in two increments: The first was 
tied to proof of enrollment at the beginning of the academic term, and the second was tied to 
proof of earning at least six credits and achieving a GPA of “C” or better. Students received re- 
minder notices from UNITE-LA about their scholarship awards and also had access to a pro- 
gram website that showed the total amount they were awarded and how much money remained 
in their accounts. This feature was particularly important for students who were awarded per- 
formance-based scholarships that extended for one or two years. (More information on the im- 
plementation of the program and how students drew down their scholarships is presented in 
Chapter 3.) 


The Higher Education Landscape in California 

As noted at the beginning of the chapter, increasing the number of college graduates is a major 
goal for California. While the state faces many challenges to achieving this goal — including 
severe budgetary and economic challenges brought on by the recession that began in 2007-2008 
— the state also benefits from an unusually strong and comprehensive postsecondary education 
system. Indeed, California’s Master Plan for Higher Education — signed into law in 1960 — 
has long been a model for the nation and other countries. It created three distinct segments: 16 

• California Community Colleges are open to all high school graduates and 
adults age 1 8 and older. The community colleges have as their primary mis- 
sion providing academic and vocational instruction through the first two 
years of undergraduate education. They are also authorized to provide reme- 
dial instruction, English as a Second Language courses, adult noncredit in- 
struction, community service courses, and workforce training courses. Cali- 
fornia’s 1 12 community colleges served over 1.2 million full-time equivalent 
students in 2010-2011. 

• California State Universities (CSUs) are open to high school graduates 
who finish in the top one-third of their class. The CSUs primarily emphasize 
undergraduate education and graduate education through the master’s degree. 

The 23 CSUs enrolled just under 288,000 full-time equivalent undergradu- 
ates in 2010-2011. 

• University of California (UC) is reserved for high school graduates who 
finish in the top one-eighth (12.5 percent) of their class. UC is designated the 
state’s primary research institution and delivers undergraduate, graduate, and 


16 Enrollment data are from Taylor (2012). 
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professional education. Nine UC campuses enrolled approximately 175,000 
full-time equivalent undergraduates in 2010-201 1. 17 

Because of this large and comprehensive system, public colleges and universities ac- 
count for the vast majority of college degrees awarded in California. Only 16 percent of the 
state’s undergraduates enroll in a private institution — the lowest rate of private college attend- 
ance among the nation’s 15 largest states. 18 

In addition to conceptualizing a postsecondary education system that would serve stu- 
dents with a wide range of academic abilities and interests, California’s Master Plan was nota- 
ble for its emphasis on affordability. For many years, the guiding principle was to provide tui- 
tion-free education to all state residents, though this policy effectively ended in the 1980s be- 
cause of state general fund reductions. Budget cuts in the early 2010s led to large increases in 
tuition and fees, though by national standards, California’s community colleges and public uni- 
versities remain relatively affordable. 19 For example, California charged $1,119 for full-time 
enrollment at a community college in 2011-2012, compared with an average of $3,288 for 
community colleges across the United States. During the same years, the cost of full-time at- 
tendance for CSU students was approximately $6,500 and for UC students was $13, 200. 20 
These figures do not include room, board, books, and ancillary expenses. 

Federal financial aid — most notably, the Pell Grant (whose maximum was $5,550 in 
2011-2012) — provides a buffer against rising tuition and fees for low-income students 
throughout the United States. 21 In addition, California has enacted its own programs to pre- 
serve the goal of tuition- free education for low-income students. For example, students attend- 
ing California community colleges may apply for the Board of Governors (BOG) Fee Waiver, 
which eliminates all enrollment fees. It has a simple application form and does not require stu- 
dents to complete the much more complicated FAFSA, though students are encouraged to do 
so in order to be considered for federal Pell Grants and other need-based aid. In recent years, 
about one-third of all California community college students received the Board of Governors’ 
Fee Waiver. 22 


17 There are 10 UC campuses, but one (University of California, San Francisco) solely enrolls graduate 
students. 

18 Johnson and Sengupta (2009). 

19 College Board (2011). 

20 See CSU and UC cost information on their respective websites (www.calstate.edu/budget and 
http :// admission, universityofcalifomia.edu). 

21 See the Office ofFederal Student Aid website (www.studentaid.ed.gov). 

22 Taylor (2012). 
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Cal Grant guarantees aid to California high school graduates and community college 
transfer students who meet specified income, academic, and other eligibility criteria. The largest 
awards (known as Cal Grant A) are reserved for students who earn at least a 3.0 GPA in high 
school or who transfer from a California community college to a baccalaureate-granting institu- 
tion. To qualify for an award under the Cal Grant program, students must complete the FAFSA 
by a spring deadline each year. In 2011-2012, over 240,000 students were awarded Cal Grants, 
with the majority of recipients enrolled in the CSUs and community colleges. 23 

In summary, the CFC-PBS evaluation took place during a time in which the higher ed- 
ucation system in California was under considerable strain. Despite the recession, the state 
managed to keep its financial aid programs intact, though the bad economy and budgetary short- 
falls hurt the state’s community colleges and universities in other ways. One likely outcome of 
the widespread publicity of financial woes is increased interest among students and their parents 
in applying for financial aid. Despite the availability of and eligibility for federal financial aid, 
many students in California have forgone applying in the past. 24 This pattern may change as 
students are now being asked to pay more for attending community colleges and state universi- 
ties than at any other time in the state’s history. 25 


Theory of Change 

Why might scholarships in general — and performance-based scholarships in particular — be 
expected to lead to better outcomes for students? A logic model depicting the theory of change 
for the CFC-PBS Program is shown in Figure 1.2. Drawing on behavioral economics and dis- 
cussions with financial aid experts, MDRC postulated that perfonnance-based scholarships 
might influence students’ thinking and behavior in several ways. 26 First, the very use of the term 
“scholarship” may confer upon students a special group status — that is, the idea of being a 
scholar who is destined for success. Such labeling may be particularly important for students 
who may not have considered themselves “star” students in the past. The benchmarks required 
to earn the money may also motivate students to attempt and complete more credits and to work 
harder at earning good grades. Second, receipt of a scholarship may help students to work fewer 

23 Taylor (2012). Awards may include full tuition coverage at the UCs and CSUs for up to four years, con- 
tributions of up to $9,708 toward tuition costs at private institutions for up to four years, and cash stipends of 
up to $1,551 to help cover books and living expenses for the neediest students. In 2011-2012, the Cal Grant 
program provided up to $5,472 to students attending CSUs and up to $12,192 to students attending UCs. These 
amounts equal the cost of tuition but do not include other fees that these institutions charge. 

24 lnstitute for College Access & Success (2010); Cochrane (2007). 

25 Taylor (2014). 

26 MDRC used guidance from the extensive literature on stereotype threat (Steele and Aronson, 1995) and 
discussions with noted cognitive psychologist Edward Deci on motivation in developing the conceptual 
framework. 
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hours and to devote more time to school. Third, receipt of a scholarship may help students cover 
other expenses associated with going to college — everything from books and supplies to trans- 
portation — and thereby increase their odds of success. 

If students respond to the scholarships in the ways outlined above, they are expected to 
meet the conditions of the performance-based scholarship and to feel increased confidence in 
their ability to succeed. They are also expected to feel less stressed about money while in col- 
lege. These short-term outcomes may lead to greater persistence in college and less time to earn 
a degree. Ultimately, these behavioral changes are expected to lead to increased graduation 
rates, better employment prospects, and the higher earnings associated with a college credential. 

While MDRC is using the logic model in Figure 1 .2 to guide the evaluation generally, 
some aspects of the model cannot be explicitly examined at every PBS Demonstration site. 27 
There are also some important questions beyond the scope of the current research. For example, 
it is difficult to know how the dire economy and the cutbacks affecting postsecondary education 
in California at the time may have influenced the students in this study. On the one hand, media 
reports of fee hikes and difficulties getting into courses could have had a chilling effect on stu- 
dents’ decision to go to college — or at least may have affected their choice of institution. On 
the other hand, it was not an easy time for recent high school graduates (or anyone else) to find 
jobs, and this may have increased the interest in attaining a degree to gain an advantage in the 
employment market while also lowering the opportunity cost for doing so. The performance- 
based scholarships were predicated on the idea that relatively modest supplements to financial 
aid would provide additional needed funding and lead to better academic outcomes, but the 
scholarships were not designed to help students deal with other, contextual problems like fewer 
course offerings at community colleges. A main advantage of the random assignment design 
used for this evaluation is that it ensures that both program and control group members experi- 
enced the same conditions, thereby providing the most reliable test of the impact of scholarships 
in difficult economic times. 

This report examines whether the different types of scholarships affect some of the in- 
trapersonal mediating factors shown in the logic model, such as motivation, and whether they 
affect short-term academic outcomes. Moreover, as this report does not analyze all terms for 
students who received the longer-duration scholarships (specifically, Scholarship Type 5 and 
Scholarship Type 6), it is premature to draw final conclusions about the effectiveness of these 
awards. Nonetheless, the early findings in this report suggest that the program was largely im- 
plemented as designed; that the intervention generated small, positive effects on enrollment; and 
that these effects were spread across numerous subgroups, though concentrated among students 


~ 7 The shaded boxes in Figure 1.2 show the aspects for which data are unavailable in the California study. 
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The Performance-Based Scholarship Demonstration 

Figure 1.2 

Logic Model for the Cash for College Performance-Based Scholarship 
Cash for College Performance-Based Scholarship Study 

Activity Mediators Outcomes 


Short Term Medium Term Long Term 



NOTE: The shaded boxes represent variables that are not examined in this report. Italics represent mediators. 













attending two-year colleges. A cross-site synthesis report will focus more heavily on longer- 
term persistence and completion of certificate and degree requirements. 


Organization of the Report 

The rest of the report is organized as follows: The research sample and evaluation design are 
described in Chapter 2. Chapter 3 covers program implementation, including students’ take-up 
of the scholarship offer. Chapter 4 contains a discussion of the effects of the scholarship pro- 
grams on students’ academic outcomes and takes a closer look at how students’ behaviors 
changed as a result of the scholarships (as determined through a follow-up survey). Chapter 5 
provides some infomiation on program costs, and Chapter 6 offers policy implications and 
conclusions. 
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Chapter 2 


Sample Intake, Sample Characteristics, 
Data Sources, and Methodology 


The California Cash for College Performance-Based Scholarship (CFC-PBS) Program is using 
a random assignment research design to estimate the effects of a performance-based scholarship 
— earned in addition to existing financial aid — on student progress toward degree attainment. 
This chapter includes a description of how students became a part of the research sample. It also 
presents a discussion of the data sources used in this report and the follow-up period for the im- 
pact analyses, gives selected characteristics of the sample members, and introduces the method- 
ology used for the analyses in this report. 


Target Population 

As noted in Chapter 1, the CFC-PBS Program is a study embedded into the statewide Cash for 
College program infrastructure. As a result, the program targeted the same population of stu- 
dents that Cash for College typically targets through its workshops: high school seniors in low- 
income communities that tend to have low college-going rates. To be eligible for the program, 
students were required to meet the following criteria: 

• Attend a Cash for College workshop in one of the four eligible regions of the 
program 

• Be a high school senior at the time of the workshop (and under the age of 20) 

• Submit a federal Free Application for Federal Student Aid (FAFSA) and a 
state Cal Grant GPA [grade point average] Verification Form by the March 2 
Cal Grant deadline 

• Complete the Cash for College Exit Survey 

• Meet low-income eligibility standards based on Cal Grant income thresh- 
olds 1 

• Sign an infomied consent form or have a parent provide consent for partici- 
pation in the study 


'For example, the Cal Grant AC income ceiling for a dependent student in a four-person household was 
$80,200 in the 2010-2011 academic year. 
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MDRC and the Cash for College partners mutually agreed to keep all eligibility re- 
quirements that had previously been established by Cash for College for receipt of the $1,000 
Cash for College scholarship (the scholarship without performance criteria). However, the in- 
come threshold and infonned consent form criteria were added for purposes of the study. 
MDRC and the Cash for College partners agreed on the income threshold after an income anal- 
ysis on workshop participants was completed. The objective of the income threshold was to en- 
sure that scholarship funds would be disbursed to students with the highest level of need. 

Under the traditional Cash for College program, students without residency status — re- 
ferred to as AB 540 students — were eligible to receive the $1,000 Cash for College scholar- 
ship. MDRC, the Cash for College partners, and the regional coordinators agreed that this past 
practice should be upheld and undocumented students should be allowed to participate in the 
study. Since AB 540 students are not eligible for federal or state financial aid, they were not 
required to complete the FAFSA and Cal Grant GPA Verification Form. Instead, they were 
asked to submit an AB 540 Income Worksheet — a form akin to the FAFSA but not submitted 
to the federal government — to determine whether they met the study’s income threshold eligi- 
bility criteria. 2 


Recruitment and Random Assignment 

Study Regions 

As mentioned above, MDRC and the Cash for College partners targeted four regions 
from which to draw the CFC-PBS sample. 3 The study used a phase-in strategy over two years to 
enroll students in the study. In 2009, the Los Angeles and Far North regions enrolled the first 
cohort of study participants. These regions were chosen for a number of reasons: 

• Both regions had the staff capacity to implement the study procedures in a 
short time frame. 

• The two regions have different student populations — Los Angeles has the 
largest minority student population, mainly Latino, while the Far North’s 
student population is mainly white. 

• Urban, suburban, and rural communities are represented across both regions. 

• There is high level of need within each region. 


2 There are 239 AB 540 students in the CFC-PBS study. 

3 In total, there were 1 1 Cash for College regional coalitions at the time of the study. 
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The Los Angeles region is primarily urban and suburban; it includes all of Los Angeles 
County and neighboring communities. The Far North region is primarily rural and is spread out 
across 1 1 counties in northern California. In 2010, Kem County and the Capital regions joined 
the study for a total of four Cash for College regions participating in the CFC-PBS study. (Refer 
to Figure 1.1 for a map of the study regions.) Similar considerations to those described above 
were used in choosing the additional regions — namely, level of need, population, and regional 
diversity. Kem County is primarily an agricultural region found in California’s Central Valley, 
while the Capital region is mostly urban and suburban. The Capital region includes Sacramento 
(the state’s capital) and surrounding communities. 

Given the study’s scale and complex design, a decision was made to enroll students 
over two years. This allowed MDRC and the Cash for College partners to address issues that 
arose early on in the implementation of the CFC-PBS Program. These lessons were then applied 
to the second round of student enrollment, which resulted in a smoother enrollment process for 
workshop organizers in the Kem County and Capital regions. 

Outreach and Recruitment 

MDRC built upon Cash for College’s publicity strategies to reach and recruit the 
study’s sample. These strategies included the “College Cash Box” that contained posters and 
information about Cal Grants — need-based entitlement and competitive awards funded by the 
state of California — and other financial aid for workshop organizers, a flyer template that 
could be customized by each site, direct mailings, and radio spots promoting Cash for College. 
In addition to letting students know they could receive help with their FAFSA by attending a 
Cash for College workshop, the promotional materials also mentioned the opportunity for stu- 
dents to be included in a drawing for a $1,000 scholarship — an incentive used by Cash for Col- 
lege to encourage students to attend workshops. 

MDRC worked with the Cash for College partners to incorporate language about the 
study and the CFC-PBS scholarships into the marketing materials provided to study regions. As 
a result of the study, students attending a Cash for College workshop could qualify for an award 
that ranged from $1,000 to $4,000. During 2009 and 2010, over 15,000 high school seniors at- 
tended a Cash for College workshop in the four study regions and expressed interest in being 
part of the study. 4 


4 For the purposes of the study, the Cash for College workshops were the vehicle to recruit the study sam- 
ple and were not a direct part of the intervention being evaluated. However, the workshops were essential in 
providing students with information about the opportunity to qualify for a CFC-PBS scholarship. 
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In addition to the marketing strategies discussed above, Cash for College conducted 
outreach to priority high schools in each region. Some of the criteria that determined a priority 
high school included a concentration of poverty, high numbers of parents with low educational 
levels, and low high school graduation rates. Workshop organizers in study regions were en- 
couraged to pay particular attention to reaching priority schools in their areas. 

Enrolling Students in the Study 

Cash for College had an existing infrastructure to provide workshop organizers with op- 
tional yearly training sessions on Cash for College workshop procedures. MDRC worked with 
the Cash for College partners to incorporate an additional training session, led by MDRC staff 
members, on study enrollment procedures. Workshop organizers were required to attend this 
training session if they wanted to offer their students the opportunity to earn a performance- 
based scholarship. The purpose of these study enrollment procedure training sessions was to 
ensure that workshop organizers, who would be explaining the study to students, understood 
what it meant for a student to be part of a random assigmnent study. It was critical for students 
— and in the case of students under age 18, for their parents — to understand this information 
clearly so that they could make an informed decision about participating in the study. 

Study training sessions were offered from December through March before study intake 
and were provided both in person and via webinars. During the sessions, workshop organizers 
learned about the time commitment and additional responsibilities required of them to effective- 
ly carry out the study procedures, which included explaining the study and obtaining students’ 
and parents’ consent to participate in the evaluation. The study training sessions also familiar- 
ized workshop organizers with the fonns students would be asked to fill out. 

Implementation of the CFC-PBS Study Procedures at the Cash for 

College Workshops 

During a typical Cash for College workshop, a financial aid specialist would go over 
each line of the FAFSA. After questions were answered, students were directed to computers to 
work on their financial aid applications. Volunteers were present during the workshop to assist 
students with any questions. Most workshops were approximately three hours long. 

A number of adjustments were made to these workshops to accommodate the study 
procedures: A fomial check-in process was put in place, after which students were given the 
study fonns and directed to a location where they were infomied about the study. 5 Staff mem- 


5 Students in the fall 2010 cohort were also shown a video in order to make sure that information about the 
CFC-PBS study was disseminated uniformly. 
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bers explained the random assigmnent process and a student’s responsibilities as a study partic- 
ipant, while emphasizing that all of the information provided would be kept confidential. 

Students who agreed to be part of the study signed an informed consent fonn and com- 
pleted an Exit Survey. Upon completing the study enrollment process, students then proceeded 
to work on their FAFSAs and seek assistance as needed. Before leaving the workshops, students 
were directed to a check-out station. A workshop organizer was responsible for ensuring that a 
student’s informed consent form was signed by the appropriate person and verifying the stu- 
dent’s age. Students who were younger than 18 and did not attend with a parent were instructed 
to have a parent or guardian sign the informed consent fonn and return it to a specific person at 
the student’s high school. Once fomis were collected, they were submitted to the appropriate 
regional coordinator. The regional coordinators packaged the paperwork and forwarded the in- 
fonned consent fonns to MDRC and the Exit Surveys and financial aid applications to the Cali- 
fornia Student Aid Commission. 

After these documents were received, MDRC verified infonned consent fonns for stu- 
dents who expressed a desire to be a part of the CFC-PBS study, while the California Student 
Aid Commission verified Exit Surveys and financial aid applications. Students were then 
matched across the three data sources, and MDRC began the process of detennining program 
eligibility. 

Overall, 15,420 students were assessed for participation, 9,665 were deemed eligible, 6 
and 5,160 participants were randomly assigned into a program group or control group stratified 
by workshop, while the remaining students were assigned to a nonstudy group. 7 Further, within 
each workshop, program group and control group students were randomly assigned to one of 
six scholarship type groups, which correspond to the six different scholarship types of varying 
amount and duration outlined in Chapter l. 8 The CFC-PBS Program’s process of random as- 
sigmnent is depicted in Figure 2.1 , 9 

After students were randomly assigned, MDRC compiled a list of scholarship recipients 
and provided the regional Cash for College coordinators with scholarship award notification 


6 That is, these students satisfied all of the bulleted criteria listed in the “Target Population'’ section pre- 
sented at the beginning of the chapter. 

7 The nonstudy group was created to minimize the size of the control group in order to lower evaluation 
costs (by lowering tracking and survey costs) while maintaining the ability to detect impacts. 

^Though control group students were randomly assigned to an individual scholarship type, they are pooled 
into a single control group in the analyses in this report. 

9 For additional detail on the exclusion and randomization process, see the Consolidated Standards of Re- 
porting Trials (CONSORT) diagram in Appendix Figure A.l and the discussion of the randomization process 
in Appendix A. 
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The Performance-Based Scholarship Demonstration 
Figure 2.1 

Random Assignment Diagram 
Cash for College Performance-Based Scholarship Study 



NOTES: a Undocumented, qualified students declare AB 540 status in order to obtain in-state tuition 
from which they would normally be precluded because of their documentation status. 

b The nonstudy group was created to limit the size of the research sample to constrain research 
costs. 
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letters. In June of 2009 and 2010, local workshop organizers notified students of their status as 
program group participants in the CFC-PBS study. Because the CFC-PBS scholarships were 
portable, students had to take active steps to become part of the program. In particular, they had 
to go to the Cash for College website and claim their scholarships by indicating which postsec- 
ondary institutions they intended to attend in the fall. Program group students who did not claim 
their awards by an early August deadline risked losing part or all of their scholarships. For a 
more in-depth discussion about the messages to CFC-PBS students after award notification, see 
Chapter 3 and Appendix Figure B. 1, which depicts a sample CFC-PBS scholarship award letter. 

MDRC also sent letters to control group students informing them that they would not be 
eligible for a scholarship, but thanking them for agreeing to participate in the study. Appendix 
Figure B.2 presents a sample CFC-PBS control group notification letter. 


Data Sources 

To examine the impact, implementation, and cost of the CFC-PBS Program, the analyses pre- 
sented in this report rely on several data sources, described below. 

Baseline Demographic Information 

• Baseline data: The Cash for College Exit Survey (also known as the Base- 
line In formation Form) provides demographic and background information 
on program and control group students before they participated in the CFC- 
PBS Program. Data from the Exit Survey are used to describe the sample and 
assess the success of random assigmnent. Students were asked to fill out the 
Exit Survey at their Cash for College workshops; those who failed to com- 
plete the survey were deemed ineligible for the study. 

Financial Aid 

• FAFSA data: The California Student Aid Commission provided financial 
aid application data for all sample members in the study. The FAFSA is an 
application filled out annually by prospective and current college students in 
order to determine eligibility for federal financial aid. Many states and post- 
secondary institutions also use the FAFSA to detennine a student’s eligibility 
for state and institution aid. 10 FAFSA data prior to random assigmnent are 


10 See the Office of Federal Student Aid’s website for more information about the FAFSA 
(http://studentaid.ed.gov). 
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available for all students in the study. 11 These data are used in this chapter to 
describe students’ financial aid application characteristics at baseline (that is, 
at the time of random assigmnent). 

• Cal Grant data: The California Student Aid Commission provided infor- 
mation on Cal Grant awards for all sample members in the study. Students 
apply for the Cal Grant by filling out the FAFSA and submitting a verified 
grade point average (GPA). If students meet the financial eligibility and GPA 
requirements, they are eligible for a Cal Grant award. Cal Grants can be used 
to pay for an array of college expenses at qualifying two-year, four-year, and 
vocational postsecondary California schools. 12 Like the FAFSA, Cal Grant 
data prior to random assigmnent are available for all students in the study. 13 
These data are used in this chapter to describe students’ financial aid award 
packages at baseline. 

Scholarship Receipt 

• Scholarship data: The California Student Aid Commission provided schol- 
arship infonnation for all program group members in the study. Findings on 
CFC-PBS scholarship payments are presented in Chapter 3 to describe pro- 
gram implementation. Scholarship receipt data are available up to three terms 
after random assigmnent. 

Enrollment 

• National Student Clearinghouse data: The National Student Clearinghouse 
(Clearinghouse), a nonprofit organization, collects and distributes enrollment, 
degree, and certificate data from over 3,300 colleges that emoll 96 percent of 
the nation’s college students. 14 Given the portable nature of the CFC-PBS 
scholarship, MDRC was not able to obtain detailed transcript infonnation for 
every school at which students were enrolled. Thus, the Clearinghouse data 
are used in Chapter 4 as the primary source of enrollment infonnation. How- 


1 'FAFSA applications are technically filed before random assignment, but it is possible that these verified 
forms may have changed after random assignment. 

12 See the California Student Aid Commission’s website for more information about the Cal Grant pro- 
gram (http://calgrants.org). 

13 Cal Grant GPA Verification Forms are also technically filed before random assignment, but it is possible 
that these verified forms may have changed after random assignment. 

14 See the National Student Clearinghouse’s website for more information 
( www. studentc learingho use. org) . 
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ever, the data do not include information about important transcript-level 
outcomes such as credits attempted, credits earned, and GPA. The Clearing- 
house data cover up to three terms after random assignment. 

• California Community College Chancellor’s Office (CCCCO) data: The 

CCCCO provided transcript data for all sample members who attended a 
California community college. Findings on California community colleges 
are presented in Chapter 4 to shed light on the effect of the performance- 
based scholarships on students enrolled in two-year schools. The CCCCO 
data cover up to two terms after random assignment. 

Survey 

• The Performance-Based Scholarship follow-up survey: MDRC designed 
a follow-up survey that was administered to all sample members in the spring 
of the year following random assignment (roughly 12 months after they at- 
tended a Cash for College workshop). The survey contained questions on a 
wide range of topics, including sample members’ educational background, 
work histories, motivation levels, social supports, health, and experiences in 
the program. Close to 80 percent of the sample completed the follow-up sur- 
vey. Student responses are presented in Chapters 3 and 4 and Appendix A. 

Field Research 

• Study partner interviews: MDRC conducted interviews with the Cash for 
College partners in the summer of 2011 to understand their perspectives on 
the implementation of the CFC-PBS Program. Findings from these inter- 
views are highlighted in Chapter 3. 

• Student focus groups: MDRC also organized student focus groups and in- 
terviewed study participants to leam about students’ experiences in college 
and perceptions of the CFC-PBS Program. Both program group and control 
group students participated in the focus groups, and students represented all 
six scholarship types. Students from both the fall 2009 and fall 2010 cohorts 
were also represented, as were community college and four-year college stu- 
dents. There were two rounds of focus groups, the first beginning in Novem- 
ber 2010 and the second beginning in May 201 1. A total of six student focus 
groups were conducted over a five-month period in the earlier round, while 
seven student focus groups and five individual student interviews were con- 
ducted over a three-month period in the latter round. Findings from these fo- 
cus groups and interviews are presented in Chapter 3. 
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Cost 

• Cost data: All costs have been categorized into one of two categories — 
scholarship payments or program administration. These two cost categories 
are estimated using different data sources. The cost of scholarship payments 
is estimated using data provided by the California Student Aid Commission. 
These costs are tracked at the student level and therefore assigned to a schol- 
arship type, which allows for a precise estimate of how much was paid out in 
scholarships for each scholarship type. The cost of program administration is 
estimated using program expenditure data as recorded by the Los Angeles 
Area Chamber of Commerce. This infomiation was tracked at the program 
level (that is, it was not tracked by scholarship type). The Los Angeles Area 
Chamber of Commerce expenditure data capture all expenditures between 
January 2009 and June 2011. A cost analysis of the CFC-PBS study is pre- 
sented in Chapter 5. 


Methodology 

The implementation research draws upon both qualitative and quantitative data sources to pro- 
vide infonnation about how the program was implemented. The qualitative data are primarily 
gathered from interviews with program staff members and focus groups with study participants 
as well as observations of program activities and operations. These sources provide impressions 
of the program from the viewpoints of those who run it and who receive its services. Quantita- 
tive data drawn from surveys and administrative records are used to understand how and how 
much of the program components were taken up, or used, by participants. 

The main analytical strategy for impacts pools all of the program groups together to 
provide an average estimate of the CFC-PBS Program by comparing the outcome for the pooled 
program groups with the outcome for the pooled control groups through standard t-tests. For 
analyses of the effect of scholarship type, the results of an overall F-test with all six program 
groups and one pooled control group are first examined. 15 If the F-test statistic of this overall 
test proves to be nonsignificant, then all observed differences between the pooled control group 
and individual program groups can be attributed to chance alone. 16 As a result, no additional 


15 See Lindquist (1953) and Chapter 3 for the underlying assumptions and Appendix A for the underlying 
calculations. 

16 Nonsignificance may also occur if the test is underpowered to detect existing differences. The proposed 
total sample size for the study was estimated to able to detect a 3.6 percentage point difference at community 
colleges and a 3.1 percentage point difference at four-year universities. For an additional discussion of power, 
see Appendix A. 
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information is gained by examining differences for individual pairs of treatments (that is, for 
individual scholarship types). This procedure effectively avoids the possibility of spurious find- 
ings as a result of multiple hypothesis testing. 17 

As a supplement to the primary analysis, MDRC analyzed transcript data obtained from 
the CCCCO to examine the effects of the scholarships for community college students. These 
data are limited to the students attending California community colleges, which is roughly half 
of the sample. MDRC takes a conservative analytical approach and assumes that students not 
attending these institutions are not enrolled in any postsecondary institution. (That is, such stu- 
dents are assumed to have zeros for outcome measures such as enrollment or credits attempted.) 


Characteristics of the CFC-PBS Sample 

The analysis sample for this report consists of 4,921 students. While 5,160 students were ran- 
domly assigned, the study’s 239 AB 540 students were excluded from all analyses because data 
collection for these students was incomplete as a result of the absence of Social Security num- 
bers and poor matches by name and date of birth. 

Table 2.1 shows the baseline characteristics from the Exit Survey for members of the 
analysis sample (hereinafter referred to simply as the sample) compared with the characteristics 
of all 2010 Cash for College workshop attendees. Around 60 percent of the CFC-PBS sample 
was female — a proportion that was fairly consistent across the four workshop regions and that 
reflects the proportion of female students who attend Cash for College workshops overall. Per 
the eligibility criteria, all sample members were high school seniors at the time of random as- 
signment. In contrast, 96 percent of all Cash for College attendees were high school seniors at 
the time of their workshops. 

Over three-fifths of sample members identified themselves as Hispanic/Latino. This 
number is about 9 percentage points higher than the proportion of all Cash for College attendees 
who identify with this ethnicity. The proportion of Latino students in the CFC-PBS sample was 
closer to three- fourths for the Los Angeles and Kern Country regions; both of these regions also 
have higher proportions of Latino students who attend Cash for College workshops overall. 18 
Around 20 percent of sample members identified as white, but this proportion ranged from 3 
percent in the Los Angeles region to over 63 percent in the Far North region. These percentages, 


17 Schochet (2008). 

18 CFC-PBS students who chose more than one race on their Exit Survey but identified as Latino were 
considered Latino, while all other Cash for College attendees who chose more than one race on their Exit Sur- 
vey were considered multiracial; this may help explain why the proportion of Latino students was higher for 
the CFC-PBS sample than for the population of all Cash for College attendees. 


29 



The Performance-Based Scholarship Demonstration 
Table 2.1 

Selected Characteristics of Sample Members and All 2010 Cash for College Attendees at Baseline, by Region 


Cash for College Performance-Based Scholarship Study 


Characteristic 

All 

CFC 

Attendees 

Study 

Analysis 

Sample 

Los Angeles 
Region- PBS 
wide Study 

Far North 
Region- PBS 
wide Study 

Kern County 
Region- PBS 
wide Study 

Capital 

Region- PBS 
wide Study 

Gender (%) 

Male 


41.3 

39.8 

41.4 

39.8 

42.1 

40.1 

41.6 

38.7 

43.3 

41.1 

Female 


58.7 

60.2 

58.6 

60.2 

57.9 

59.9 

58.4 

61.3 

56.7 

58.9 

Race/ethnicity d (%) 

Latino 


52.1 

60.9 

73.6 

77.4 

14.7 

22.5 

66.2 

73.9 

29.0 

40.9 

White 


20.0 

20.3 

4.4 

2.9 

63.8 

63.6 

17.6 

13.6 

16.3 

12.8 

Black 


5.4 

3.8 

6.0 

4.6 

1.0 

0.8 

5.0 

4.0 

14.1 

9.8 

Asian or Pacific Islander 


13.1 

11.0 

11.3 

13.2 

5.9 

5.3 

4.4 

5.0 

25.8 

28.5 

Other 


9.4 

4.0 

4.8 

2.0 

14.6 

7.8 

6.9 

3.5 

14.9 

8.0 

High school senior (%) 


96.5 

100.0 

97.3 

100.0 

99.8 

100.0 

99.6 

100.0 

96.7 

100.0 

First person in family to attend college (%) 


49.4 

53.7 

59.7 

60.7 

33.3 

38.6 

52.3 

55.9 

46.7 

48.1 

Highest degree/diploma earned by either parent (%) 
Not a high school graduate 

31.8 

34.9 

45.3 

45.1 

12.6 

13.5 

39.1 

37.6 

27.8 

26.6 

High school diploma or GED certificate 


27.5 

31.1 

27.1 

29.9 

27.4 

29.8 

30.1 

36.3 

33.6 

36.9 

Some college or associate’s degree 


23.1 

22.7 

17.5 

17.1 

32.4 

35.8 

19.3 

19.5 

26.9 

25.3 

Bachelor’s degree or higher 


17.7 

11.3 

10.2 

8.0 

27.5 

20.8 

11.6 

6.6 

11.8 

11.2 

Language other than English regularly 

spoken at home (%) 


55.3 

60.7 

76.0 

80.5 

17.2 

18.3 

55.7 

61.3 

49.3 

54.5 

High school cumulative GPA (average) 


- 

2.9 

- 

2.9 

- 

3.0 

- 

2.8 

- 

3.0 


(continued) 



Table 2.1 (continued) 


Characteristic (%) 

All 

CFC 

Attendees 

Study 

Analysis 

Sample 

Los Angeles 
Region- PBS 
wide Study 

Far North 
Region- PBS 
wide Study 

Kern County 
Region- PBS 
wide Study 

Capital 

Region- PBS 
wide Study 

Motivation to apply for financial aid b (average) 
Relative Autonomy Index 


1.4 


1.5 


1.1 


1.6 


1.7 

External regulation subscale 

- 

6.7 

- 

6.8 

- 

6.7 

- 

6.7 

- 

6.7 

Introjected regulation subscale 

- 

4.2 

- 

4.2 

- 

4.1 

- 

4.1 

- 

4.0 

Identified regulation subscale 

- 

6.3 

- 

6.3 

- 

6.2 

- 

6.3 

- 

6.3 

Integrated regulation subscale 

- 

6.4 

- 

6.5 

- 

6.2 

- 

6.4 

- 

6.5 

Sample size 

25,860 

4,921 

5,423 

2,760 

2,114 

1,263 

2,064 

622 

1,023 

276 


SOURCES: MDRC calculations using Exit Survey (Baseline Information Form) data and Cal Grant data provided by the California 
Student Aid Commission, and Exit Survey data compiled by the California Student Aid Commission for the statewide population. 

NOTES: Estimates are adjusted by research cohort. 

Missing values are not included in individual variable distributions. 

Distributions may not sum to 100 percent because of rounding. 

a For the columns that present PBS study measures, students who identify as Latino are shown only in the Latino category, even if they 
selected more than one race. Students who selected more than one race and are not Latino are considered multiracial. The “other” category 
comprises American Indian/ Alaska Native or multiracial students, or students of some other race/ethnicity. For the columns that present 
region-wide measures, the same categories apply except that students who chose more than one race are considered multiracial, regardless 
of whether they identify as Latino. 

b Motivation to apply for financial aid is defined using the Relative Autonomy Index (RAI), which has a range of -18 to 18, where a 
higher value represents greater autonomous motivation. The RAI is calculated as a weighted average: RAI = [Externalx(-2)] + 
[Introjectedx(-l)] + [Identifiedx(l)] + [lntegratedx(2)j. See Deci, Koestner, and Ryan (2001). 



both total and by region, are consistent with the proportions of white students who attend Cash 
for College workshops overall. 

Over half of the CFC-PBS sample reported that they were the first member of their 
family to attend college. In fact, only one-third of sample members have a parent with any col- 
lege-going experience. CFC-PBS students were less likely to have family members with some 
college education — both overall and by region — than were all Cash for College workshop 
attendees. 

Over 60 percent of sample members reported speaking a language other than English 
with high frequency at home, ranging from a low of 18 percent in the Far North region to a high 
of over 80 percent in the Los Angeles region. Again, these proportions were slightly higher for 
CFC-PBS students than for all workshop attendees. 

Given the growing literature on the possibility for incentives to undermine intrinsic mo- 
tivation for completing tasks, the Relative Autonomy Index (RAI) provides an overall measure 
of whether CFC-PBS students’ motivation to apply for financial aid was internally driven. 19 The 
average RAI for the full CFC-PBS sample was 1.4, which indicates a relatively low level of 
intrinsic motivation to apply for financial aid, meaning that few students were inherently inter- 
ested in applying for aid but were likely compelled more by external factors (the scale ranges 
from -18 to 18). Internal motivation to apply for financial aid was highest in the Capital region 
(1.7) and lowest in the Far North region (1.1). The Los Angeles and Kern County regions had 
RAI values slightly higher than the mean, but no region demonstrated a particularly high level 
of intrinsic motivation. 

On balance, the study evaluates an intervention targeted to a low-income sample with a 
sizable proportion of minority students — a sample that the California Student Aid Commission 
does a successful job of targeting through its Cash for College workshops. Though there are a 
few characteristics on which the CFC-PBS sample and 2010 Cash for College workshop at- 
tendees differ — namely, the proportions of Latino students, first-time college-goers, and non- 
native English speakers — by and large, these two populations are quite similar. 

It is important to bear in mind that the CFC-PBS Program sample was not designed to 
be representative of all California first-year students. It was drawn from a population of students 
who actively chose to attend a Cash for College workshop and opted to be considered for the 
study. This self-selecting group of fairly motivated students was drawn while students were still 
in high school applying for admission to college. 


19 For more information about the RAI see www.selfdeterminationtheory.org/self-regulation- 
questionnaires. 
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Table 2.2 shows the same baseline characteristics from the Exit Survey for sample 
members, but presented by research group. That is, the second column of Table 2.1 is identical 
to the first column of Table 2.2. Of the 4,921 students in the CFC-PBS analysis sample, one- 
third were randomly assigned to the program group, and the rest were assigned to the control 
group. An asterisk in the far-right column of Table 2.2 indicates that the percentage point differ- 
ence between the program and control groups for a given characteristic is statistically significant 
at a prespecified level. Statistical significance indicates that there is only a small probability that 
the observed difference between the two groups occurred by chance. (See Box 2. 1 on how to 
read an impact table.) Overall, the proportions for each baseline characteristic are almost identi- 
cal for the program and control groups. The few differences that are observed are no more than 
what would be expected to occur by chance. 20 

Baseline financial aid data were also collected for all students in the sample through the 
FAFSA and Cal Grant GPA Verification Fomi, both of which were completed about three to 
five months before random assignment. The first panel of Table 2.3 shows that nearly all stu- 
dents in the CFC-PBS study completed their FAFSA. 21 Over 95 percent of sample members 
were dependents of their parents, with an average household size of four members. The Adjust- 
ed Gross Income (AGI) was around $34,000 for both program and control group students, and 
around half of sample members received no public benefits, while the other half received most- 
ly food stamps or free or reduced-price lunches. 

The Expected Family Contribution (EFC) reflects the amount of money students are 
expected to pay out of pocket or procure in additional loans to cover the Cost of Attendance 
(COA). The EFC, which is not expressed in dollars, is calculated based on family income and 
size, state of residence, and a number of other factors. In general, a lower EFC is associated 
with being eligible for higher levels of need-based aid. 22 The average EFC for students in the 
CFC-PBS study was around 2,200, with 52 percent of the sample having an EFC of zero. Over 
86 percent of both program and control group students had EFCs that made them Pell-eligible. 

The second panel of Table 2.3 shows that virtually all program and control group mem- 
bers completed their Cal Grant GPA Verification Fonns. Around 20 percent of students were 
eligible for and awarded Cal Grant A, while around 44 percent were instead eligible for and 
awarded Cal Grant B. No program or control group members qualified for Cal Grant C, which 


20 A p-value of 0.57 was reported for the F-test of whether baseline characteristics jointly predict research 
group status. Convention suggests that this probability is large enough that the potential differences can be ig- 
nored in the analyses. See Appendix A for a discussion of the employed methodology. 

21 Because of application errors, MDRC did not obtain verified, post-random assignment FAFSA data for 
10 students or verified, post-random assignment Cal Grant data for five students. 

22 U.S. Department of Education, Federal Student Aid, Customer Experience Group (2011). 


33 



Box 2.1 


How to Read the Impact Tables in This Report 

Most tables in this report use the format illustrated in the table excerpt below, which displays 
a hypothetical enrollment outcome for the program and control groups. The one row of data 
shows that 83.9 percent of program group students and 80.4 percent of control group students 
enrolled in any college during their first term in the study. 

The “Difference” column in the table shows the observed difference between the two research 
groups on the outcome — that is, the estimated average impact of the opportunity to partici- 
pate in the program. For example, the estimated average impact on enrollment can be calcu- 
lated by subtracting 80.4 from 83.9, yielding an impact estimate of 3.5 percentage points. 

Differences marked with one asterisk or more are considered statistically significant, meaning 
that there is a low probability that the difference occurred by chance. Differences that have no 
asterisk indicate that the opportunity to participate in the program did not have a discernible 
effect on that outcome. Assuming the true effect is zero, the number of asterisks indicates the 
probability that an estimate at least as large as the observed difference could have occurred by 
chance. One asterisk corresponds to a 10 percent probability, two asterisks to a 5 percent 
probability, and three asterisks to a 1 percent probability. The more asterisks that appear next 
to a positive difference, the more likely it is that the opportunity to participate in the program 
had a true positive average impact on the outcome. The impact in the table exceipt below has 
two asterisks, indicating that the impact is statistically significant at the 5 percent level — 
meaning that there is a 5 percent chance of observing an estimated average impact this large 
(or larger) if the opportunity to participate in the program actually had no average effect on 
first-term enrollment. In other words, there is a 95 percent level of confidence that the oppor- 
tunity to participate in the program had a positive impact on first-term enrollment. 

Also shown in the table is the standard error of the impact estimate. The standard error is a 
measure of uncertainty or variability around the impact estimate. A useful guideline is that the 
confidence interval is usually calculated as 1 .96 multiplied by the standard error (for a 95 per- 
cent confidence interval). In the example below, the confidence interval is 2.4 (1.96 multi- 
plied by the standard error, or 1 .96 x 1 .2). Thus, there is a 95 percent chance that the “true” 
average impact on enrollment lies between 1.1 percentage points and 5.9 percentage points, 
calculated as 3.5 ± (1.96 x 1.2). 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

First program term 





Enrolled in any college 

83.9 

80.4 

3.5 ** 

1.2 
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The Performance-Based Scholarship Demonstration 
Table 2.2 

Selected Characteristics of Sample Members at Baseline, by Research Group 


Cash for College Performance-Based Scholarship Study 



Analysis 

Program 

Control 

Characteristic 

Sample 

Group 

Group 

Gender (%) 




Male 

39.8 

39.3 

40.1 

Female 

60.2 

60.7 

59.9 

Race/ethnicity a (%) 




Latino 

60.9 

60.9 

60.9 

White 

20.3 

19.9 

20.5 

Black 

3.8 

3.4 

4.0 

Asian or Pacific Islander 

11.0 

10.8 

11.1 

Other 

4.0 

5.0 

3.5 ** 

High school senior (%) 

100.0 

100.0 

100.0 

First person in family to attend college (%) 

53.7 

54.5 

53.3 

Highest degree/diploma earned by either parent (%) 




Not a high school graduate 

34.9 

35.1 

34.7 

High school diploma or GED certificate 

31.1 

29.8 

31.8 

Some college or associate’s degree 

22.7 

24.0 

22.1 

Bachelor’s degree or higher 

11.3 

11.0 

11.5 

Language other than English regularly spoken at home (%) 

60.7 

59.9 

61.0 

High school cumulative GPA (average) 

2.9 

2.9 

2.9 

Motivation to apply for financial aid b (average) 




Relative Autonomy Index 

1.4 

1.5 

1.4 

External regulation subscale 

6.7 

6.7 

6.8 ** 

Introjected regulation subscale 

4.2 

4.2 

4.2 

Identified regulation subscale 

6.3 

6.3 

6.3 

Integrated regulation subscale 

6.4 

6.4 

6.4 

Sample size 

4,921 

1,640 

3,281 

(continued) 
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Table 2.2 (continued) 


SOURCES: MDRC calculations using Exit Survey (Baseline Information Form) data and Cal Grant 
data provided by the California Student Aid Commission. 

NOTES: To analyze whether baseline characteristics jointly predicted research group status, a 
likelihood ratio test was performed, which yielded a p-value of 0.57. This suggests that the differences 
in baseline characteristics between program and control group students are likely to have occurred by 
chance. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Missing values are not included in individual variable distributions. 

Distributions may not sum to 100 percent because of rounding. 

a Students who identify as Latino are shown only in the Latino category, even if they selected more 
than one race. Students who selected more than one race and are not Latino are considered multiracial. 
The “other” category comprises American Indian/ Alaska Native or multiracial students, or students of 
some other race/ethnicity. 

b Motivation to apply for financial aid is defined using the Relative Autonomy Index (RAI), which 
has a range of -18 to 18, where a higher value represents greater autonomous motivation. The RAI is 
calculated as a weighted average: RAI = [Externalx(-2)j + [Introjectedx(-l)] + [Identifiedx(l)] + 
[Integratedx(2)j. See Deci, Koestner, and Ryan (2001). 


can be used to help pay for tuition and expenses at occupational or technical colleges for up to 
two years. As discussed in Chapter 1 , Cal Grant A awards can be used to help cover tuition and 
fees at public and private institutions for up to four years; for CSU and UC schools, this 
amounts to contributions of up to $4,429 and $1 1,124, respectively. For private institutions, this 
amounts to annual awards of up to $9,708. Cal Grant B awards provide the neediest students 
with a stipend of up to $1,551 to help cover books and living expenses; after a student’s first 
year, Cal Grant B awards also help cover tuition and fees at public and private institutions in the 
same amount as Cal Grant A awards. 23 The average amount of financial aid awarded from the 
Cal Grant was around $2,400 for both program and control group students. 

Because part of determining Cal Grant eligibility is verifying a student’s GPA, Table 
2.3 also shows the average high school GPA for all students in the sample. 24 Around 48 percent 
of program and control group students reported a high school GPA of 3.0 to 4.0, the GPA range 


23 Though no CFC-PBS students were awarded a Cal Grant C, students who are enrolled in vocational 
programs that are not based at California community colleges are also eligible to receive up to $2,592 for tui- 
tion and fees. See the California Student Aid Commission’s website (www.calgrants.org) for additional infor- 
mation. 

_4 ln order to qualify for Cal Grant A, students must have earned at least a 3.0 GPA in high school or have 
transferred from a California community college to a baccalaureate-granting institution. Cal Grant B awards 
are reserved for the neediest students who maintained at least a 2.0 GPA in high school. 
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The Perfomance-Based Scholarship Demonstration 
Table 2.3 

Financial Aid Application Characteristics of Sample Members at Baseline 
Cash for College Performance-Based Scholarship Study 


Characteristic 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Free Application for Federal Student Aid (FAFSA) 

Completed the FAFSA 3 (%) 

99.8 

99.8 

0.0 

0.1 

Dependency status (%) 

Dependent 

95.8 

95.8 

0.0 

0.6 

Independent 

4.2 

4.2 

0.0 

0.6 

Household size (%) 

1 

3.4 

3.2 

0.2 

0.5 

2 

9.8 

10.1 

-0.2 

0.9 

3 or 4 

47.3 

46.8 

0.5 

1.5 

5 or more 

39.5 

40.0 

-0.5 

1.5 

Average household size 

4.2 

4.2 

0.0 

0.0 

Average total number of household members in 

college, including sample member 

1.3 

1.3 

0.0 

0.0 

Received public benefits' 5 (%) 

SSI 

4.6 

5.1 

-0.5 

0.7 

Food stamps 

9.1 

10.6 

-1.5 * 

0.9 

Free or reduced-price lunch 

42.3 

43.4 

-1.1 

1.4 

TANF 

1.8 

1.9 

-0.2 

0.4 

WIC 

5.9 

6.0 

-0.1 

0.7 

No benefits 

53.7 

52.6 

1.1 

1.4 

Adjusted Gross Income (AGI) (%) 

Less than $10,000 

10.4 

11.3 

-0.9 

1.0 

$10,000 to less than $20,000 

19.0 

19.3 

-0.3 

1.2 

$20,000 to less than $30,000 

19.9 

18.5 

1.4 

1.2 

$30,000 to less than $40,000 

14.2 

15.2 

-1.0 

1.1 

$40,000 to less than $50,000 

12.4 

12.6 

-0.2 

1.0 

$50,000 or more 

24.1 

23.0 

1.0 

1.3 

Average Adjusted Gross Income ($) 

34,184 

33,749 

435 

728 


(continued) 
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Table 2.3 (continued) 


Characteristic 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Expected Family Contribution (EFC) (%) 

0 

53.9 

53.9 

0.0 

1.5 

1 to less than 2,500 

20.9 

20.8 

0.1 

1.2 

2,500 to less than 5,000 

11.3 

12.5 

-1.2 

1.0 

5,000 or more 

13.9 

12.8 

1.1 

1.0 

Average Expected Family Contribution (EFC) 

2,256 

2,136 

121 

163 

Pell-eligible L (%) 

86.3 

87.2 

-0.9 

1.0 

Cal Grant 

Applied for Cal Grant 2 (%) 

99.9 

99.9 

0.1 

0.1 

Awarded Cal Grant (%) 

64.1 

63.6 

0.4 

1.4 

Awarded Cal Grant type A 

20.2 

19.4 

0.8 

1.2 

Awarded Cal Grant type B 

43.9 

44.2 

-0.3 

1.5 

Average Cal Grant award ($) 

Average Cal Grant award among recipients 

2,407 

3,754 

2,394 

3,765 

13 

100 

High school GPA (%) 

3.0 to 4.0 

48.3 

47.8 

0.5 

1.5 

2.0 to 2.9 

43.6 

44.4 

-0.8 

1.5 

Less than 2.0 

8.1 

7.8 

0.3 

0.8 

Sample size (total = 4,921) 

1,640 

3,281 




(continued) 
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Table 2.3 (continued) 

SOURCES: MDRC calculations using FAFSA and Cal Grant data provided by the California Student 
Aid Commission. 

NOTES: To analyze whether baseline characteristics jointly predicted research group status, 
likelihood ratio tests were performed on FAFSA characteristics and on Cal Grant characteristics. For 
FAFSA characteristics, this test yielded a p-value greater than 0.99. For Cal Grant characteristics, this 
test yielded a p-value greater than 0.99. Convention suggests that these probabilities are large enough 
that these potential differences can be ignored in the analyses. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Characteristics shown in italics are calculated for a portion of the full sample, and indicate 
nonexperimental data. 

Distributions may not sum to 100 percent because of rounding. 

a MDRC and CSAC ensured that students completed the FAFSA and Cal Grant GPA Verification 
Form prior to enrolling in the study, but because of application errors, MDRC did not obtain verified 
FAFSA data for 10 students or verified Cal Grant data for 5 students. 

b Distributions may not sum to 100 percent because categories are not mutually exclusive. SSI is 
Supplemental Security Income. TANF is Temporary Assistance for Needy Families. W1C is the 
Supplemental Nutrition Program for Women, Infants, and Children. 

C MDRC calculated Pell eligibility using the maximum EFC eligible for a Pell Grant. Maximum 
EFC for Pell eligibility was 4,617 for 2009-2010 and 5,273 for 2010-2011. This measure represents a 
proxy for Pell eligibility and does not reflect whether students actually received a Pell Grant. 


needed to be eligible for a Cal Grant A, while a slightly lower proportion reported a high 
school GPA of 2.0 to 2.9. 25 

Together, the FAFSA and Cal Grant characteristics indicate that the sample is made up 
primarily of low-income, dependent students with high levels of financial need. The relatively 
high proportion of students with a GPA of 3.0 to 4.0 supports the notion that the CFC-PBS 
sample is a self-selecting population of fairly motivated students. As was the case with the data 
shown in Table 2.2, the baseline financial aid characteristics of program and control group 
members are virtually identical. The few differences that do exist between research groups are 
no more than what would be expected to occur by chance. 26 


25 The Cal Grant B cutoff is a 2.0 GPA. While the proportion awarded Cal Grants may be low given the 
sample eligibility criteria, the 36 percent not awarded can be broken down into 8 percent not meeting the GPA 
requirement and 13 percent exceeding the Cal Grant B income limits but failing to meet the Cal Grant A GPA 
requirement. The remaining 15 percent may be related to Cal Grant application processing procedures and 
other reasons. Chapter 4 explores the connection between Cal Grant receipt and CFC-PBS awards. 

-6 A p-value of 0.99 was reported for the F-test of whether FAFSA application characteristics jointly predict 
research group status. A p-value of 0.99 was reported for the F-test of whether Cal Grant characteristics jointly 
predict research group status. Convention suggests that both of these probabilities are large enough that the poten- 
tial differences can be ignored in the analyses. See Appendix A for a discussion of the methodology employed. 
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Chapter 3 

Program Implementation 


This chapter describes how performance-based scholarships were incorporated into the Cash for 
College program. It also discusses student participation rates in the Cash for College Perfor- 
mance-Based Scholarship (CFC-PBS) Program and presents student accounts regarding their 
experiences with the program and how they used their scholarship dollars. 

The following are among the key findings: 

• The existing structure of the Cash for College program provided a strong 
foundation that facilitated the development of the CFC-PBS Program, which 
was largely implemented as designed. 

• Over 80 percent of students received at least one scholarship payment over 
the first two tenns and the initial enrollment payment of the third tenn. Stu- 
dents who received Scholarship Type 6 ($4,000 over two years) were eligible 
for and received the most money on average, while students who received 
Scholarship Type 3 ($1,000 over one year) received the least money on aver- 
age. Because complete data are not yet analyzed for students receiving 
Scholarship Type 5 ($2,000 over two years) and Scholarship Type 6 ($4,000 
over two years), it is too early to determine whether the payment patterns 
among the different scholarship types are significant. 

• Based on qualitative research, the scholarships’ grade point average (GPA) 
benchmark of “C” or better seemed attainable to large numbers of students. 
However, knowing that they could lose the award did appear to give a con- 
siderable number of students an incentive to work harder, based on inter- 
views and survey results. 

• Students who received CFC-PBS payments reported that the money was 
primarily used for college-related expenses — books being the most com- 
monly purchased item. They also mentioned that the timing of the disburse- 
ment of CFC-PBS moneys helped cover up-front costs that were incurred be- 
fore financial aid was disbursed. 
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Program Operations: Embedding Performance-Based 
Scholarships into Cash for College 

The CFC-PBS Program is unique within the PBS Demonstration in that California’s program 
was implemented within an existing framework and operated not by a single college or univer- 
sity but by a partnership of two principal organizations: the California Student Aid Commission 
(a government agency) and the Los Angeles Area Chamber of Commerce (L.A. Chamber, a 
private organization). 

Cash for College Partner Roles 

The California Student Aid Commission was responsible for the administration of the 
CFC-PBS Program’s notification and claiming procedures. Additionally, it oversaw the web- 
based management system used for collecting student data and administrative records and for 
tracking payment documentation. These responsibilities were in addition to the California Stu- 
dent Aid Commission’s significant role in the outreach and recruitment of study participants for 
the CFC-PBS study (as discussed in Chapter 2). 

The Los Angeles Chamber Foundation (an affiliate of the L.A. Chamber), the fiscal 
agent for Cash for College, oversees the day-to-day operation of the verification and disburse- 
ment procedures of the traditional Cash for College scholarship. For the purposes of the CFC- 
PBS study, it also assumed the administration of the CFC-PBS Program scholarships. 

Scholarship Policies and Procedures 

MDRC worked closely with the Cash for College partners to adapt existing scholarship 
notification, claiming, verification, and disbursement processes for the purposes of the study. 
The most significant modifications were made to the verification and disbursement procedures. 
The addition of a performance payment for the CFC-PBS Program required that some major 
enhancements be made to the Cash for College management database. In order to effectively 
implement the six scholarship types of the CFC-PBS Program, the database needed to have the 
capacity to track different cohorts, types of payments (enrollment and performance), and multi- 
ple verification points, in addition to meeting the data needs of the study. 

In keeping with past Cash for College program practices, it was incumbent upon stu- 
dents to submit the required documentation (class schedules or transcripts) to verily they met 
the scholarships’ enrollment or performance criteria. No changes were made to the verification 
and disbursement of the traditional Cash for College scholarship (Scholarship Type 1 — $1,000 
over one term with no performance incentive), including the award being paid to students’ insti- 
tutions. The CFC-PBS Program’s five performance-based scholarships (Scholarship Types 2 
through 6) had two payments: an enrollment payment (similar to the Cash For College scholar- 
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ship), contingent upon enrollment in six credit hours or more at an accredited, degree-granting 
institution in the United States, and a performance payment, contingent on students’ completing 
six credits or more with a “C” or better GPA. Fall-term payments for all five performance-based 
scholarships were paid in two equal installments per term. The spring payment for Scholarship 
Types 3 through 6 (offered over one or two academic years) had no enrollment payment and 
consisted of one performance payment disbursed at the end of the term. See Table 3.1 for more 
details on the semester payment points and payment amounts for each scholarship type. 1 

MDRC and the Cash for College partners wanted to create as strong an incentive as 
possible for students to enroll at least part time, perform satisfactorily, and receive all the money 
that was offered. Hence, some flexibility was put in place in the following ways: 

• Students receiving the two-year scholarships (Scholarship Types 5 and 6) did 
not have to enroll consecutively; they could “stop out” for a semester and 
still remain eligible for the remaining portion of their scholarships. In other 
words, students who attended the first term and not the second, but returned 
to school in the third term of a four-term scholarship, would still be eligible 
for the scholarship payments in the third and fourth terms as long as they met 
the enrollment and perfonnance benchmarks in those terms. However, these 
students would forfeit the second-term award. 

• Students who failed to meet the scholarship’s criteria for a given semester of 
the program still had the opportunity to receive the following semester’s 
payment. In essence, their slate was wiped clean; the “C” or better GPA re- 
quired for the scholarships was based on each semester’s GPA rather than 
cumulative perfonnance. 

• While students were strongly encouraged to observe document submission 
deadlines, the program did not penalize students for submitting documents 
past the deadline; students still remained eligible for their scholarships. 2 

Notifying Scholarship Recipients 

As mentioned in Chapter 2, the Cash for College workshops, the primary function of 
the Cash for College program, served as the outreach mechanism to enroll both program and 
control group students into the CFC-PBS study. However, an important distinction should be 


'For simplicity of discussion, only the semester payment structure is referenced in the text. However, Ap- 
pendix Table C.l shows the payment schedule adapted for students who attended institutions operating on a 
quarter system. 

2 ln practice, 368 of the 1,640 program group members in the study sample (or 22.4 percent) submitted 
their documents after the program’s submission deadlines for one or more payments. 
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The Performance-Based Scholarship Demonstration 
Table 3.1 

Semester Payment Schedule 


Cash for College Performance-Based Scholarship Study 



Scholarship 

Scholarship 

Scholarship 

Scholarship 

Scholarship 

Scholarship 

Characteristic 

Type 1 

Type 2 

Type 3 

Type 4 

Type 5 

Type 6 

Performance-based scholarship 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Amount of scholarship per semester ($) 

1,000 

1,000 

500 

1,000 

500 

1,000 

Duration of scholarship 

1 semester 

1 semester 

2 semesters 

2 semesters 

4 semesters 

4 semesters 

First year 
Fall payments ($) 
Enrollment 

1,000 

500 

250 

500 

250 

500 

Performance 

- 

500 

250 

500 

250 

500 

Spring payment ($) 

Performance 

— 

— 

500 

1,000 

500 

1,000 

Second year 
Fall payments ($) 
Enrollment 





250 

500 

Performance 

- 

- 

- 

- 

250 

500 

Spring payment ($) 

Performance 

— 


— 

— 

500 

1,000 

Total scholarship amount ($) 

1,000 

1,000 

1,000 

2,000 

2,000 

4,000 


NOTE: For simplicity, the payment schedule for semester-based institutions is shown. The majority of the students in the CFC-PBS study 
attended semester-based institutions. 



made between the Cash for College workshops and the CFC-PBS Program. Students’ recruit- 
ment into the CFC-PBS Program was initiated through the workshops, but participation in the 
study began after they were randomly assigned into one of the two study groups, the program 
group or control group. Thus, although they were essential for the recruitment and intake of the 
students, the workshops themselves are not a direct part of the intervention being tested. 

As also noted in Chapter 2, MDRC built upon Cash for College’s scholarship selection 
process to randomly assign program group students to one of the study’s six scholarship types. 
The notification letters sent to program group students congratulated them on their award and 
fonnally inducted them into the “Cash for College Scholars Group” (that is, the program 
group). In addition to making the students feel part of a special group, the letter served as a 
means to inform students of the scholarship type they were awarded and provide instructions on 
the process for claiming the scholarship payment via the Cash for College scholarship-claiming 
website. Along with the scholarship award letter, a handout (shown in Figure B.l) was includ- 
ed, which was intended to do the following: 

• Provide students with more detailed instructions on how to log on to the 
Cash for College scholarship-claiming website 

• Remind students that their scholarship is perfonnance-based 

• Give students an overview of the enrollment and perfonnance award verifi- 
cation and payment process 

• Alert students that the CFC-PBS Scholars website (described below) would 
be the primary means of communication regarding their particular scholar- 
ship 

• Reiterate to students that they were part of a select group 

Scholarship Claiming 

Following standard Cash for College practices, students awarded CFC-PBS Program 
scholarships were required to complete two steps in order to receive the scholarships. The first 
step was to declare their intent to enroll in an accredited college or university in the fall after 
attending a Cash for College workshop. Students did so by accessing the Cash for College 
scholarship-claiming website and filling out a form using the infomiation provided in their noti- 
fication letter. The site was linked to a web-based management infonnation system that was 
used by the Cash for College program and the L.A. Chamber to collect student data and track 
verification documentation and scholarship disbursements. 
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After claiming their scholarship awards, CFC-PBS Program scholarship recipients were 
redirected from the Cash for College scholarship-claiming website to the CFC-PBS Scholars 
website, which was developed solely for study participants. Here students found a customized 
home page with specific infomiation about their scholarship — how much they had received, a 
calendar of submission deadlines, and other pertinent infonnation about their award. 

In addition to directing students to access infomiation about their scholarships from the 
Scholars website, MDRC built upon the Cash for College electronic reminder system, which 
consisted of e-mails sent to Cash for College scholarship recipients reminding them to submit a 
class schedule by the given deadline. E-mails modeled after the Cash for College electronic re- 
minders were developed for the CFC-PBS Program. These reminded scholarship recipients of 
the class schedule or transcript they needed to submit to receive either an enrollment or perfor- 
mance payment. Reminder e-mails were sent out once before each verification deadline. In 
keeping with past practice, students who did not claim their scholarships by the given deadline 
received one phone call as a reminder. 

The CFC-PBS Program’s communication components, described above, were imple- 
mented during the early period of the CFC-PBS Program and were modeled after those used by 
the Cash for College program. Communication with scholarship recipients was envisioned to be 
limited (or “light-touch”). In addition, the CFC-PBS model is unlike others in the PBS Demon- 
stration in that students were expected to assume the bulk of the responsibility for turning in the 
required documents in order to receive their scholarships. 


Scholarship Payments 

Enrollment Payment 

Once students completed the first step in the scholarship-claiming process, meaning 
they went to the Cash for College scholarship-claiming website to claim their payments, they 
were expected to follow through with the second step in order to receive their scholarship pay- 
ments: Submit the appropriate verification documents to the L.A. Chamber. Class schedules 
were used to determine whether students had met the scholarships’ enrollment criteria. Once 
staff members at the L.A. Chamber confinned that each class schedule came from an accredited 
institution and the student met the necessary credit hours, they would authorize the disburse- 
ment of the initial enrollment payment. 

Performance Payment 

CFC-PBS Program Scholarship Types 2 through 6 had a second payment based on per- 
formance criteria. Students had a choice of submitting either an unofficial or official transcript 
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to verify they had met the CFC-PBS Program academic benchmark (a “C” or better GPA) in six 
or more credit hours. Documentation could be submitted via e-mail, fax, or regular mail. Once 
all the information was verified by L.A. Chamber staff members, perfonnance payments were 
authorized for disbursement. 

Disbursement 

During the study period, the Los Angeles Chamber Foundation continued to send the 
traditional Cash for College scholarship payments directly to students’ institutions by check, as 
was the prior practice. However, using paper checks for payment disbursements of the other 
scholarship types was an inefficient option, as the CFC-PBS Program brought about an increase 
in the numbers of scholarships awarded, variations in amount, additional payment points over 
multiple terms, and awards being paid directly to individual students with shifting contact ad- 
dresses (as opposed to single institutions). Moving to an electronic transfer system by which 
direct deposits were made to student bank accounts was seen as the best way to effectively 
manage the payment process. During the claiming process, students were encouraged to sign up 
for direct deposit, and instructions were posted on the Scholars website. The electronic funds 
transfer payment option applied only to Scholarship Types 2 through 6. The majority of stu- 
dents opted to have their scholarship moneys deposited directly into their bank accounts. The 
option to request a check was still available to students, though only 12 percent of scholarship 
recipients chose this option. 

Another change in the scholarship administration process resulting from the implemen- 
tation of the CFC-PBS Program was that L.A. Chamber staff members had to ensure the au- 
thenticity of hundreds of class schedules and transcripts from 170 different colleges and univer- 
sities for each disbursement period. The biggest challenge of verifying whether students met 
enrollment and perfonnance benchmarks was the variation among transcripts from different 
institutions. In some cases, L.A. Chamber staff members had to make manual credit and GPA 
conversions to understand whether a student met the scholarship’s requirements. 3 

Spot Audit 

Because documents could be submitted by the students electronically, they were easier 
to alter compared with official documents sent directly from a college or university. Thus, there 
was a concern that students might submit fraudulent documentation to receive a particular 
scholarship payment. To address this concern, MDRC and the Cash for College partners estab- 


3 Some colleges and universities count units for developmental courses in a student’s overall GPA, while 
others do not. The CFC-PBS Program did count developmental courses toward the scholarship benchmarks. In 
addition, some institutions give cumulative, not term, GPAs. 
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lished an auditing process that was communicated to students. The hope was that this process 
would deter any student from altering verification documents. During the verification process, 
every twenty- fifth student who submitted an unofficial transcript was asked to submit an official 
one. The official document would be checked against the unofficial copy to ensure the authen- 
ticity of the information before payment was released. 4 

Evolution of the CFC-PBS Program's Student Communication 

Unlike the other models in the PBS Demonstration, the CFC-PBS Program was forced 
to confront a communication challenge. The portable nature of the scholarships resulted in 
1,720 program group students being spread out among 170 colleges and universities. The CFC- 
PBS Program could thus not rely on a campus-based representative or institution resources 
(both of which were available to the other PBS Demonstration interventions) to ensure scholar- 
ship recipients continued to be engaged with the program for the duration of their scholarships. 

As the CFC-PBS Program began, MDRC and the Cash for College partners questioned 
whether the light-touch communication strategy would be enough to keep the CFC-PBS Pro- 
gram at the forefront of students’ minds. Subsequently, a decision was made to enhance the stu- 
dent communication components of the program. In addition to providing infonnation about the 
processes students had to follow in order to receive their scholarships, the enhancements were 
meant to engage scholarship recipients to create a sense of community among students as well 
as a more direct connection to the CFC-PBS Program. The use of online media and electronic 
communication was thought to be the best strategy to accomplish these goals. 

To develop an effective messaging strategy, MDRC retained the services of a behavior- 
al economist through a program sponsored by the An nie E. Casey Foundation. The consultant 
worked with the Cash for College partners and MDRC to develop messages that used concepts 
from behavioral economics in an effort to increase student engagement. One recommended 
strategy was to create a strong “scholar” group identity tied to the CFC-PBS Program. From the 
consultant’s perspective, this had the potential to prime students to remember the scholarship 
and its motivating power. Additionally, it could subtly convey to students that they could 
achieve and persist in college. Targeting the program’s messages was believed to increase the 
chances that students would exhibit the changes in behavior that the CFC-PBS Program was 
designed to trigger: taking more classes, spending more time studying, and strengthening self- 
perception. 


4 Over the course of the study, only one student was found by the auditing process to have submitted sus- 
pect documentation. Upon discovery, the student was asked to submit an official transcript but never responded 
to the request. No further disbursements were made to the student. 
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The Scholars website was enhanced with affirming statements throughout. Students 
were referred to as “Cash for College Scholars” on the website’s home page, which was person- 
alized for each student. In addition to scholarship information, a dial was added that provided 
each student with a visual representation of his or her scholarship earnings in real time. Infor- 
mation about scholarships, motivational facts, and short videos with advice from other CFC- 
PBS Program scholarship recipients were also added to the website. Finally, a scholar advice 
section was developed. Students could submit questions and receive an answer from a staff 
member who monitored questions and responded. Motivational e-mails that reinforced the 
group’s identity — “Cash for College Scholars” — and encouraged students to work hard in 
order to earn their scholarships were sent to students in addition to e-mails reminding them of 
deadline submissions. Text messages to remind students of upcoming submission deadlines 
were also added in the second year of the program. 


Was the Program Implemented as Designed? 

The CFC-PBS study was built on a solid foundation: a strong partnership and a sound pro- 
grammatic framework. As is common with organizations involved in many new programs, the 
Cash for College partners, made up of a small number of staff members from each partner or- 
ganization, underwent some growing pains early in the study’s implementation. As mentioned 
previously, the study required the enhancement and addition of procedures to manage the vol- 
ume of students receiving awards and the disbursement of the various CFC-PBS scholarship 
payments. The study also required that additional, more complex data be collected and shared 
between organizations, something that had been done on a much smaller scale in prior years. 
These requirements caused initial hurdles, primarily related to technical issues and new account- 
ing procedures and systems, but these impediments were eventually overcome. 

Overall, the implementation story suggests that the CFC-PBS Program received a “fair 
test.” In other words, the impacts found in this research should reflect the program itself and not 
any flaws in implementation. Changes made to Cash for College’s notification, claiming, verifi- 
cation, and disbursement systems to create the CFC-PBS Program were implemented largely as 
designed. 

However, MDRC and the Cash for College partners’ venture into online media and 
electronic communication turned out to be too ambitious. MDRC and the Cash for College 
partners put significant thought into the challenges that a portable scholarship might face in 
terms of connecting students to the program. The Scholars website was envisioned as the prima- 
ry mechanism that would keep the scholarship and program at the forefront of students’ minds. 
However, an analysis of website usage data showed that for the most part, the website was un- 
derused by students. 
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A number of issues affected the ability of the additional communication elements to 
live up to their full potential. MDRC and the Cash for College partners did not have the ability 
to maintain and update the Scholars website in a timely manner, and as a result, students’ en- 
gagement with it diminished over time. In addition, several limitations related to maintaining 
the confidentiality of the study’s subjects precluded the addition of more interactive elements to 
the various communication tools. Ultimately, the Scholars website was considered too expen- 
sive relative to its reach or impact on students, and it was discontinued in the fall of 201 1. 

Student Participation in the Cash for College Performance-Based 

Scholarship Program 

The proportion of students in the program group who received a scholarship measures 
the level of participation in the program. Program participation is defined as the rate at which 
students met the academic benchmark and submitted their transcript verification information. 
Thus, students who are identified as having received a scholarship are those who enrolled for or 
completed six or more credits with a “C” or better GPA and submitted the necessary paperwork 
to receive a scholarship. 

Table 3.2 presents the scholarship receipt rates for students in the program group and 
for each scholarship type, from the first term through the enrollment payment of the third term. 
Because students receiving Scholarship Type 5 ($2,000 over two years) and Scholarship Type 6 
($4,000 over two years) are eligible to receive payments over four temis, it is too early to draw 
meaningful conclusions from comparisons of take-up rates across scholarship types, as this re- 
port covers just the first three tenns. 

The panel listing first-term outcomes in Table 3.2 shows that over 93 percent of pro- 
gram group members claimed their scholarships by going to the Cash for College scholarship- 
claiming website and verifying their institution information. While over 82 percent of all pro- 
gram group students received their first-term enrollment payments, this percentage ranged from 
a low of 79.6 percent for Scholarship Type 1 students to a high of nearly 85 percent for Scholar- 
ship Type 6 students. According to enrollment data from the National Student Clearinghouse 
(which are discussed in greater detail in Chapter 4), the proportion of program group students 
enrolled in any postsecondary institution in the first program term ranged from a low of around 
88 percent for Scholarship Type 1 students to a high of over 90 percent for Scholarship Type 6 
students. Thus, most students who did not receive a scholarship payment in the first term failed 
to enroll in any college. However, a higher proportion of students enrolled (89.3 percent in the 
first term, shown in Table 4.1) than received a payment (82.6 percent in the first temi), indicat- 
ing that a small contingent of program group students (over 6 percent) did not receive a scholar- 
ship payment, either because they were enrolled for fewer than six credits or because they did 
not submit their transcript verification information. 
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The Performance-Based Scholarship Demonstration 
Table 3.2 

Scholarship Receipt Among Program Group Members: First Through Third Terms 


Cash for College Performance-Based Scholarship Study 


Outcome 

Program 

Group 

1 

2 

Scholarship Type 

3 4 

5 

6 

Performance-based scholarship 


No 

Yes 

Yes 

Yes 

Yes 

Yes 

Amount of scholarship per semester ($) 


1,000 

1,000 

500 

1,000 

500 

1,000 

Duration of scholarship 


1 semester 

1 semester 

2 semesters 

2 semesters 

4 semesters 

4 semesters 

First term 

Claimed scholarship (%) 

93.4 

91.8 

91.7 

94.5 

94.6 

92.4 

95.6 

Received a scholarship payment (%) 

82.6 

79.6 

79.9 

83.2 

84.4 

83.3 

84.9 

Received performance payment 11 

60.0 

NA 

54.5 

61.5 

62.3 

60.1 

61.0 

Average scholarship amount received ($) 

624 

796 

672 

374 

770 

382 

749 

Average scholarship amount among recipients 

756 

1,000 

841 

450 

912 

458 

882 

Second term 

Received a scholarship payment 1 ’ (%) 

45.5 

NA 

NA 

40.3 

44.2 

50.0 

47.4 

Average scholarship amount received ($) 

317 

NA 

NA 

186 

408 

228 

446 

Average scholarship amount among recipients 

697 

NA 

NA 

462 

924 

456 

941 


(continued) 




Table 3.2 (continued) 


Outcome 

Program 

Group 

Scholarship 
Type 1 

Scholarship 
Type 2 

Scholarship 
Type 3 

Scholarship 
Type 4 

Scholarship 
Type 5 

Scholarship 
Type 6 

Third term 

Received a scholarship payment (%) 

56.8 

NA 

NA 

NA 

NA 

56.2 

57.4 

Average scholarship amount received ($) 

196 

NA 

NA 

NA 

NA 

127 

267 

Average scholarship amount among recipients 

346 

NA 

NA 

NA 

NA 

226 

465 

Cumulative terms 1 through 3 

Received one or more scholarship payments (%) 

83.2 

79.6 

79.9 

83.2 

85.1 

84.4 

86.8 

Average scholarship amount received ($) 

902 

796 

672 

560 

1,178 

737 

1,462 

Average scholarship amount among recipients 

1,084 

1,000 

841 

674 

1,384 

873 

1,685 

Proportion of possible award received (%) 

63.2 

79.6 

67.2 

56.0 

58.9 

59.0 

58.5 

Sample size 

1,640 

279 

264 

273 

276 

276 

272 


SOURCES: MDRC calculations using scholarship payment data provided by the California Student Aid Commission as well as data from the 
U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary Education Data System (IPEDS). 

NOTES: Characteristics shown in italics are calculated for a portion of the program group, and indicate nonexperimental data. NA = not 
applicable. At the time of analysis, the “received any scholarship payment’’ row reflects initial enrollment payments only and not performance 
payments. 

Students assigned to Scholarship Type 1 are eligible for an enrollment scholarship payment only in the first program term, and thus are 
excluded from the performance scholarship payment in the first term and from all measures in all other terms. 

Students assigned to Scholarship Type 2 are eligible for scholarship payments only in the first program term, and thus are excluded from all 
measures from the second program term onward. 

Students assigned to Scholarship Types 3 and 4 are eligible for scholarship payments only in the first and second program terms, and thus 
are excluded from all measures from the third program term onward. 

Third program term average scholarship amounts do not include performance payments as fall 2010 cohort data were not available at the 
time of data acquisition. Cumulative average scholarship amounts also exclude these awards. 

a Includes performance payments made at quarter institutions during the winter term, which represent a small portion of the sample. 

b All scholarship payments in the second program term were performance payments. 



Program group students with Scholarship Types 2 through 6 were eligible to receive a 
performance payment in the fall tenn, and 60 percent of these students met their perfonnance 
benchmarks and submitted verification documents. 5 Scholarship Type 2 students received this 
first-term perfomiance payment at a rate of 54.5 percent, which was around 6 percentage points 
to 8 percentage points lower than rates attained by Scholarship Types 3 through 6 students. 
Overall, there was about a 23 percentage point drop in scholarship receipt from the initial en- 
rollment payment to performance payment for all Scholarship Type 2 through 6 program group 
students. The magnitude of the decline in scholarship receipt from enrollment to perfonnance 
payment in the CFC-PBS study was very close to that at the PBS New York and Opening 
Doors Louisiana sites, at which site coordinators disbursed scholarship payments based on in- 
house transcript infonnation. 6 This is encouraging, given that program group students in the 
CFC-PBS study are required to submit their transcript infonnation manually, so the decline is 
not likely to be a result of this potential hurdle. 

In the second tenn — in which only one payment was disbursed at the end of the tenn 
— only program group students with Scholarship Types 3 through 6 were eligible to receive 
perfonnance payments. Table 3.2 indicates that close to 46 percent of these students received 
their second-tenn perfonnance payments. Scholarship Type 5 and 6 students received the 
scholarship at slightly higher rates than Scholarship Type 3 and 4 students. Enrollment rates 
(shown in Table 4.1 in the following chapter) do not vary dramatically between the first and 
second tenns, and thus do not explain the drop in scholarship receipt from the first program 
tenn to the second. The overall percentage of program group students receiving a perfonnance 
payment in the second tenn is slightly lower for the students in the CFC-PBS study than for 
those in the Opening Doors Louisiana study. The perfonnance-based scholarship payment rate 
for Opening Doors Louisiana in the second program tenn was around 58 percent. It may be 
possible that part of the decline is due to students failing to submit their paperwork. 7 

Only Scholarship Type 5 and 6 students were eligible for third-term payments. About 
56 percent of these students received their initial th ir d-term enrollment payments. There was 


5 This percentage includes Scholarship Type 3 through 6 students who attended quarter-type institutions 
and received a performance payment in the winter quarter. Around 14 percent of Scholarship Type 3 through 6 
students received a performance payment in the winter quarter. 

6 See Patel and Rudd (2012) and Richburg-Hayes et al. (2009). The magnitudes of the declines in scholar- 
ship receipt from the first-term enrollment payment to the first-term performance payment in the New York 
and Louisiana studies were 27 percent and 22 percent, respectively. Additionally, recruitment for the CFC-PBS 
study occurred among students in their senior year of high school, rather than among students already regis- 
tered at their site’s college. Students in the New York and Louisiana studies also represent a nontraditional 
college-going sample, with a high proportion of older female students with at least one child. 

7 See Chapter 4 for an estimate of the proportion of students at community colleges who may have failed to 
submit their paperwork. See Richburg-Hayes et al. (2009) for additional details. 
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little difference in the proportion of Scholarship Type 5 and 6 students receiving the third-term 
enrollment payment. 

The final panel of Table 3.2 summarizes the first three terms for the full program group 
and six scholarship types. Over 83 percent of all students received a scholarship in at least one 
of the program tenns. The average amount received from the first term through the enrollment 
payment of the third term was just over $900 for all program group students. 

Scholarship Type 6 students had the potential to receive the most scholarship money, 
and they did indeed receive the most money on average, at around $1,462. That is, Scholarship 
Type 6 students received an average of $1,462 over the first two tenns and the enrollment pay- 
ment of the third tenn. Given that Scholarship Type 6 students were eligible to receive $2,500 
by the third-term enrollment payment, these students received around 58 percent of the maxi- 
mum total scholarship available to them. Scholarship Type 3 students received the least money 
on average over the first two tenns and the enrollment payment of the third term, at $560, or 
around 56 percent of the maximum total scholarship available ($1,000). Given that complete 
data for Scholarship Type 5 and 6 students are not analyzed in this report, it is too early to de- 
termine whether payment patterns among the different scholarship types are stable. 


Student Perceptions and Experiences 

MDRC conducted a follow-up survey, 13 focus groups, and a small number of individual stu- 
dent interviews to complement the study’s quantitative data sources. Analysis of the qualitative 
data collected from CFC-PBS study participants allowed field researchers to capture the stu- 
dents’ impressions of the program’s significance. Additionally, these data provided a picture of 
student participation in the program as well as a glimpse into how students used their CFC-PBS 
awards. 


This section provides a brief description of who participated in the focus groups and in- 
terviews and the external context that existed during students’ participation in the program. This 
is followed by what was learned from CFC-PBS Program scholarship recipients about their ex- 
periences participating in the program and how they used their scholarship dollars. 

Focus Group Participants 

As noted in Chapter 2, MDRC field researchers conducted 13 student focus groups and 
5 individual interviews. The focus groups involved 43 program and control students attending 
higher education institutions in the Far North, Central Valley, and Los Angeles regions and 
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San Francisco. 8 In addition, five students spoke with an MDRC field researcher in individual 
interviews. 

Students who attended the focus groups and participated in individual interviews are not 
representative of the overall study sample. In fact, a few indicators suggest that this group may 
be more motivated than the average student in the sample. For example, sizable numbers of fo- 
cus group participants took part in specialized programs, such as the Advanced Via Individual 
Determination (AVID) program, a college preparatory program that operates in many high 
schools. Students also took the initiative to attend the focus groups when asked by MDRC staff 
members, whom they had never met and to whom they had little connection through the opera- 
tion of the program. 

The Higher Education Landscape in California 

These focus groups and interviews were conducted in the midst of an economic reces- 
sion and during a period of severe budget cutbacks to higher education in California (as dis- 
cussed in Chapter 1). Students in both the program group and control group shared that they 
were beginning to feel the effects of California’s economic climate, especially as it related to the 
cutbacks in higher education. Across all institutions that focus group participants attended, stu- 
dents reported experiencing a reduction in class offerings (a direct result of budget cuts) and 
expressed having a harder time registering for the necessary classes to progress toward their 
educational goals in a timely manner. In addition, several community college students felt they 
would be more affected by the budget cuts when they were ready to transfer to a four-year uni- 
versity. Students attending San Francisco State University noted that they were already bracing 
for a 30 percent tuition increase, supporting the fears of the community college students on the 
transfer path. 

Students also mentioned experiencing changes in registration policies, which they felt 
compounded the problem of registering for necessary classes. Students in two different focus 
groups cited college registration policies that prevented them from registering for a full-time 
course load during early registration; one institution had an eight-unit cap on early registration, 
while the other had a five- unit cap. 9 Students expressed frustration at being forced to scramble 
during colleges’ add period at the beginning of the semester to register for the remaining classes 
they needed for a full-time course load or completion of their majors. Some students mentioned 
that certain classes needed by large numbers of students were already full by the time they tried 

8 The names of students invited to participate in focus groups and individual interviews were not selected at 
random from the full study sample. Students were selected from colleges that had high numbers of program 
and control group members attending them. 

9 The registration policies were cited by students attending a community college and students attending a 
state university. 
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to add them to their schedules. In some cases, students had to take unnecessary classes in order 
to continue to qualify for financial aid. 

It is important to remember that both program group students and control group stu- 
dents experienced the same conditions described above. While program group students were 
eligible to receive extra money on top of their financial aid, this eligibility did not grant them 
any other special privileges during registration. 

Meeting the PBS Academic Benchmark 

As discussed in the first section of this chapter, in order for students to receive their 
scholarships, they had to submit a transcript for verification that showed they had met the “C” 
or better GPA benchmark in six credit hours or more. Overall, program group students who par- 
ticipated in focus groups or interviews indicated they were not concerned about meeting the 
academic criteria. Their lack of concern stemmed from the perception that the GPA benchmark 
set by the CFC-PBS Program was low. Data from the follow-up survey appear to be mixed on 
whether a larger proportion of program group students shared this lack of concern. These data 
are presented in Table 3.3. About 12 percent of 1,394 program group respondents reported that 
meeting the benchmarks was difficult or very difficult, and about 36 percent reported that meet- 
ing the benchmarks was “a little” difficult. An additional 36 percent reported no difficulty at all. 
It is worth noting that almost three-fourths of the program group felt that the benchmarks were 
not at all difficult or a little difficult, but only 60 percent of students submitted documentation 
that they had met the performance benchmark in the first tenn (not shown in the table). 

When asked, “Did the scholarship encourage you to work harder?” almost 75 percent of 
1,394 program group respondents from the follow-up survey indicated that it did, while 22 per- 
cent reported no effect, and slightly more than 3 percent reported that the scholarship pushed 
them to take easier classes. One student attending a community college in the California Central 
Valley reported that the scholarship helped him focus on keeping his grades up. Even when 
more difficult courses caused his grades to dip, he said, the requirements of the PBS scholarship 
prompted him to study more and work harder to meet the benchmarks. 

When asked whether the program encouraged students to take more classes, around 43 
percent of 1,394 program group respondents reported that it did, while 54 percent reported that 
the scholarship had no effect on the number of classes they took. About 3 percent of respond- 
ents reported taking fewer classes. About 1 1 percent of program group respondents reported 
that the program influenced class selection in some other way. 

Overall, the survey suggests that the program may have had some positive effect on 
course taking and increased effort among program group members. However, the majority of 
focus group participants mentioned that while the extra money was beneficial, they did not feel 
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The Performance-Based Scholarship Demonstration 
Table 3.3 

PBS Survey Responses: Program Group Experiences 


Cash for College Performance-Based Scholarship Study 


Outcome (%) 

Program 

Group 

Understood conditions necessary to receive payment 

87.4 

CFC-PBS encouraged you to: 

Work harder 

74.6 

Take easier courses 

3.4 

No effect 

21.9 

Take more courses 

42.6 

Take fewer courses 

3.1 

No effect 

54.4 

CFC-PBS influenced course selection in some other way 

11.2 

Difficulty in meeting CFC-PBS academic benchmark 

Not difficult at all 

36.4 

A little difficult 

36.2 

Difficult 

8.1 

Very difficult 

3.4 

Missing 

15.9 

Payment schedule most preferred from CFC-PBS 

Half at the beginning and half at the end 

46.3 

One large sum at the end of the term 

15.9 

Smaller amounts each time, but paid more frequently 

15.4 

Some other schedule 

6.4 

Missing 

15.9 

Sample size 

1,394 


SOURCE: MDRC calculations using responses from the Performance-Based Scholarship 12-Month 
Survey. 

NOTES: Missing values are included only in variable distributions for characteristics with more 
than 5 percent of the sample missing. 

Distributions may not sum to 100 percent because of rounding. 


the scholarship by itself made a significant difference in pushing them academically. As men- 
tioned earlier, focus group participants were a more motivated group of students; thus, the per- 
ception that the scholarship did not influence academic behavior could reflect the types of stu- 
dents who self-selected to participate in the focus groups. 
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Communication with the Program 


Program group students who were currently enrolled or had ever enrolled in a postsec- 
ondary institution were asked if they understood the conditions they needed to meet and the 
processes they needed to follow in order to receive their scholarship. Of 1,394 respondents, 87 
percent responded affirmatively. 10 This finding appears to support the idea that the original lim- 
ited communication strategy — a website with scholarship infonnation, one reminder e-mail 
per verification point, and a contact phone number and e-mail — may have been sufficient to 
get large numbers of students to turn in the documents needed to receive their scholarships. 

In contrast, infonnation from the focus groups, interviews, and follow-up survey seems 
to indicate that the additional enhancements made to the various communication tools to keep 
students engaged — a resource-rich website embedded with motivational language, motivation- 
al e-mails, and texting — were not effective. For example, even though the Scholars website 
was embedded with motivational messaging and enhanced with additional resources beyond 
infonnation related to the CFC-PBS Program verification and submission procedures, students 
reported that they did not take the time to explore other pages on the website. They primarily 
used the site for scholarship verification infonnation, which was its original purpose. 

Payments and Use of Scholarship Money 

Large numbers of students in both the survey and focus groups reported using their 
scholarship money for college-related expenses; books were the most commonly purchased item. 
As shown in Table 3.4, the majority (over 89 percent) of fall 2010 cohort program group students 
who recalled receiving payments from CFC-PBS reported using the scholarship money to pur- 
chase books and supplies, with the next largest percentage (50 percent) reporting expenditures on 
tuition and fees. Other examples of college-related purchases and expenses cited by focus group 
participants include laptops, equipment fees, and parking pennits. One student shared that the 
scholarship helped her in a significant way — she was able to pay her tuition by the payment 
deadline, which prevented her from being dropped from her classes. Additionally, she stated, 
“[The PBS scholarship money] makes me feel that I can go to school. It took away stress.” She 
also recognized that the funds filled the gap “when [her] other money ran out.” In addition to 
school-related expenses, a smaller percentage of students reported using scholarship money to 
cover basic necessities such as transportation, food, bills, child care, or other expenses. 


10 However, this does not necessarily imply that students truly understood the criteria. In Angrist, 
Oreopoulos, and Williams (2010), when researchers asked students to state the eligibility requirements in the 
initial survey, about one-third reported the criteria incorrectly. 
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The Performance-Based Scholarship Demonstration 
Table 3.4 

PBS Survey Responses: Scholarship Use Among the Program Group 
Cash for College Performance-Based Scholarship Study 


Outcome 

Question 

Sample 

Size 

Percentage 

of 

Respondents 

Recall being a participant in performance-based scholarship study 

1,393 

84.2 

Among those who recall being a participant in the 
performance-based scholarship study: 

Remember receiving payment from CFC-PBS a 

651 

86.6 

Among those who remember receiving payment from CFC-PBS: 
Scholarship was used: b 

To purchase books and supplies 

564 

89.5 

To help with tuition and fees 

564 

49.5 

To pay for transportation 

564 

39.0 

To buy food 

564 

33.0 

To pay bills 

564 

25.0 

To deposit into bank account 

564 

26.2 

To buy clothes for self 

564 

14.2 

For entertainment or to buy something not normally afforded 

564 

8.0 

To work fewer hours at job 

564 

3.5 

To help with child-care costs, buy clothes/shoes for children or 

other family members 

564 

2.8 

For some other reason 

564 

1.1 

Main use of scholarship: " 

To purchase books and supplies 

564 

64.9 

To help with tuition and fees 

564 

17.9 

To pay for transportation 

564 

6.7 

To buy food 

564 

1.8 

To pay bills 

564 

4.8 

To deposit into bank account 

564 

2.5 

To buy clothes for self 

564 

0.2 

For entertainment or to buy something not normally afforded 

564 

0.4 

To work fewer hours at job 

564 

0.0 

To help with child-care costs, buy clothes/shoes for children or 

other family members 

564 

0.0 

For some other reason 

564 

0.9 

Sample size 

1,394 



(continued) 
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Table 3.4 (continued) 

SOURCE: MDRC calculations using responses from the Performance-Based Scholarship 12-Month 
Survey. 

NOTES: Characteristics shown in italics are calculated for a portion of the program group respondents 
and indicate nonexperimental data. 

Missing values are not included in individual variable distributions. 

Distributions may not sum to 100 percent because of rounding. 

“Question was asked of fall 2010 cohort students only. 

b Distributions may not sum to 100 percent because categories are not mutually exclusive. 


In addition to discussing what the scholarship money was used for, focus group stu- 
dents also raised the issue of the timing of the scholarship disbursements. Students in all focus 
groups reported on the inconvenient timing of financial aid disbursements, which typically oc- 
cur a few weeks into the academic tenn. Students with scholarships lasting more than one aca- 
demic tenn noted that the CFC-PBS payments were very timely, as the money helped cover 
some up-front costs until other sources of financial aid became available. This was especially 
true after the first academic tenn; money from an end-of-the-tenn performance payment could 
go toward expenses incurred at the beginning of the following semester. 

The follow-up survey also asked students about the timing of the CFC-PBS disburse- 
ments, shown in Table 3.3. Of 1,394 survey respondents, 46 percent preferred receiving two 
equal payments per tenn (one at the begimiing and one at the end), while 16 percent leaned to- 
ward receiving one lump sum at the end of the tenn. 

Finally, for some students, the CFC-PBS scholarships meant much more than just mon- 
ey to cover college-related expenses. A student who received a Scholarship Type 6 ($4,000 over 
two years) took the time to send an e-mail in which he shared the impact the scholarship has had 
on his academic journey and beyond: 

You cannot believe how much help being a part of this study has been over the 
last two years. At times I was worried that my financial aid would not be enough 
to cover the cost of my books for the semester, but this scholarship was always 
there to alleviate the stress that comes with paying for college.... Thank you and 
your staff for all of your support. You all have definitely changed my life for the 
better. 


Conclusion 

In sum, the Cash for College program components provided a solid foundation for the overlay 
of the CFC-PBS Program. Data from qualitative interviews with the Cash for College partners, 
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the follow-up survey, and student focus groups support the conclusion that the CFC-PBS Pro- 
gram was largely carried out as envisioned. 

Furthermore, survey and qualitative data shed light on some of the experiences of CFC- 
PBS scholarship recipients. Significant numbers of students mentioned not being worried about 
meeting the program’s “C” or better GPA benchmark. However, while students did not appear 
concerned about the academic benchmark, they reported that the additional money did push 
them to work harder academically. Additionally, for many students, the supplemental financial 
aid provided by the performance-based scholarships allowed them to cover up-front college- 
related expenses, the cost of textbooks in particular. 

More than 80 percent of students received at least one scholarship payment during the 
first two-and-a-half terms. Those students who did not receive a scholarship payment may have 
never enrolled or submitted verification documentation, dropped out, or been unable to receive 
a scholarship payment in any term. Scholarship payment patterns do seem to differ slightly 
among scholarship types, but because Scholarship Type 5 and 6 students are eligible for pay- 
ment in the fourth term, it is not possible to interpret whether these differences in payment rates 
are meaningful using the early data presented in this report. 

The next chapter examines the effects of the program on educational and socio- 
psychological outcomes for all students in the study. 
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Chapter 4 

Overall Effects on Academic and Other Outcomes 


While there are a handful of studies on the effectiveness of financial incentives for postsecond- 
ary students that have employed rigorous random assignment designs, the Cash for College Per- 
formance-Based Scholarship (CFC-PBS) Program is unique in that it compares the effective- 
ness of need-based aid with that of similar aid with performance criteria, and it also permits an 
examination of whether effectiveness varies by the amount and duration of the financial incen- 
tives. 1 This chapter analyzes the effect of the CFC-PBS Program on enrollment and other edu- 
cational outcomes up to three terms after students enrolled in the study. The chapter focuses on 
three primary research questions: 

• Does the CFC-PBS Program affect matriculation and enrollment? 

• Does the program work better for a particular group of students? 

• Does the intervention affect student behavior? For example, does the pro- 
gram induce students to exert greater effort in their studies or does it affect 
their motivation to do well academically? 

The key findings are as follows: 

• The CFC-PBS Program generated a modest positive impact on first-term en- 
rollment of about 5 percentage points above the control group average en- 
rollment rate of 84.4 percent. It produced somewhat smaller positive effects 
on second- and third-term enrollment. These effects extended to numerous 
subgroups, such as males, females, and students of Hispanic/Latino ethnicity. 

The program had stronger effects for those students who may not have been 
intrinsically motivated to apply for financial aid, meaning that some external 
factor (such as pressure from parents) was more responsible for a student’s 
participation. There is strong evidence that the program affected students 
with lower high school grade point averages (GPAs) more than students with 
higher high school GPAs. 

• Matriculation impacts were concentrated among students attending two-year 
colleges (specifically, California community colleges). This is reasonable 


'See Chapter 6 for a summary of research on the effect of financial aid on academic achievement. 
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given the timing of notification of scholarship eligibility in June (the summer 
prior to fall matriculation). The program also induced students attending Cal- 
ifornia community colleges to attempt and to earn a greater number of col- 
lege-level credits. However, the increase in credits attempted and earned is 
relatively small in magnitude (about one-quarter of a course on average). 

• The CFC-PBS Program seems to have increased the “good” types of extrin- 
sic motivation. That is, the survey findings suggest that the money from the 
intervention likely motivated students to achieve academically, and the moti- 
vation may have remained when the stimulus of the money was removed. 

• There is some evidence that the intervention changed academic effort, based 
on the proxy measures included in the survey, and there is also evidence that 
the intervention decreased employment. 

The remainder of this chapter proceeds as follows: First, the effect of the program on 
early academic outcomes is provided and the findings by various subgroups are presented. Next, 
possible reasons for the findings are explored. The chapter concludes with a summary of the 
lessons learned. 


Early Academic Outcomes 

The CFC-PBS Program provides a need-based grant to sample members contingent on their 
enrolling and earning grades of a “C” or better in at least six credits. Each term starts anew, so 
payments are contingent on current academic performance and not previous performance. 
These rules, along with the time-limited nature of eligibility, provide a strong incentive for pro- 
gram group members to meet the academic benchmarks in the early terms of their eligibility. 
Thus, impacts on enrollment, credits attempted, and credits earned are expected to occur fairly 
early in the period after students were randomly assigned to different scholarship groups. If 
those receiving the performance-based scholarships persist and earn more credits than they 
would have in the absence of the program, the impacts could translate into graduation effects 
over time. On the other hand, if the main effect is to increase the number of credits earned for 
those who eventually would persist and earn additional credits anyway, then the program im- 
pacts will tend to dissipate over time as control group members “catch up.” Prior research sug- 
gests that either effect can occur, but impacts on persistence, credits attempted, and credits 
earned are reasonable during the terms the program operates . 2 The main analyses examine ma- 


2 For example, Brock and Richburg-Hayes (2006), Cha and Patel (2010), and Patel and Rudd (2012) all 
find impacts on credits attempted and credits earned during the terms in which program group students remain 
eligible for performance-based scholarships. Only Brock and Richburg-Hayes (2006) find sizable impacts on 


64 



triculation and persistence rates for the Cash for College (CFC) program group eligible for the 
need-based scholarship (Scholarship Type 1, $1,000 without performance criteria) and the per- 
formance-based scholarships (PBS) program group (eligible for Scholarship Types 2 through 6) 
compared with the control group. 3 Additional sensitivity analyses are limited to the PBS pro- 
gram group and the control group. 

Matriculation 

The first panel in Table 4. 1 shows impacts on enrollment in the first term after high 
school completion using administrative data from the National Student Clearinghouse (Clear- 
inghouse). 4 The first column shows the CFC program group, or the group that was randomly 
assigned to receive the CFC $1,000 scholarship that is purely need-based (Scholarship Type 1). 
The second column of the table shows the PBS program group, or the larger group of students 
randomly assigned to receive one of the five performance-based scholarship types (Scholarship 
Types 2 through 6). About 88 percent of CFC program group students and 89 percent of PBS 
program group students enrolled in college, compared with about 84 percent of control group 
students, for a difference of 3.5 and 4.9 percentage points, respectively. This latter impact is sta- 
tistically significant, indicating that the increase in matriculation is not likely to have occurred 
by chance. 5 These high rates of enrollment are not surprising given the target sample for the 
CFC-PBS Program — that is, high school seniors (most with their parents) who committed to 
spend a day to complete the Free Application for Federal Student Aid (FAFSA). The last set of 
columns shows the difference in matriculation rates for the two program groups. The difference 
in enrollment between the CFC program group and PBS program group is 1.4 percentage 
points. This difference is statistically insignificant, which implies that the impact estimates are 
not distinguishable from each other. 


persistence and Patel, Richburg-Hayes, de la Campa, and Rudd (2013) find impacts on graduation rates after 
the program ends. Taken together, there is mixed evidence on whether performance-based scholarships are 
more likely to accelerate the accumulation of academic outcomes faster than what would have occurred with- 
out the interventions. See Patel, Richburg-Hayes, de la Campa, and Rudd (2013) for a summary of the interim 
findings from performance-based scholarships. 

3 The distinction between these two groups is made here in order to more succinctly compare students eli- 
gible for Scholarship Type 1 with students eligible for Scholarship Types 2 through 6. 

4 While most of the students in the study were found in the Clearinghouse database, 320 students — or 6.5 
percent of the observations — were not found. See Appendix A for details. 

5 The impact for the CFC program group of 3.5 is barely insignificant at the 10 percent level (p-value = 
0.103). When including other covariates that may improve precision, such as high school GPA, gender, 
race/ethnicity, highest degree earned by parent, and so on, the impact on first-term enrollment becomes signifi- 
cant (p-value = 0.082). However, the standard errors of the remaining outcomes only marginally decrease, so 
the improvement in precision does not qualitatively change the story depicted in Table 4.1. 
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The Performance-Based Scholarship Demonstration 
Table 4.1 

Enrollment Outcomes: First Through Third Terms 


Cash for College Performance-Based Scholarship Study 


Outcome (%) 

Average Outcome Levels 
CFC PBS 

Program Program Control 
Group Group Group 

CFC Program 
vs. Control 

Standard 
Difference Error 

PBS Program 
vs. Control 

Standard 
Difference Error 

PBS Program 
vs. CFC Program 
Standard 
Difference Error 

First term 

Enrolled in any college 3 

87.9 

89.3 

84.4 

3.5 

2.1 

4 *** 

1.1 

1.4 

2.3 

Enrolled in any 2-year college 

48.3 

47.9 

43.2 

5.2 

* 3.1 

4 7 *** 

1.6 

-0.5 

3.2 

Enrolled in a California community college 

48.0 

47.5 

42.7 

5.3 

* 3.1 

4 g *** 

1.6 

-0.5 

3.2 

Enrolled in any 4-year college 

39.9 

42.8 

42.8 

-2.9 

3.1 

0.0 

1.6 

3.0 

3.2 

Enrolled in a California State University college 

22.6 

23.1 

22.9 

-0.3 

2.6 

0.2 

1.4 

0.5 

2.8 

Enrolled in a University of California college 

11.2 

13.3 

12.9 

- 1.8 

2.1 

0.3 

1.1 

2.1 

2.2 

Second term 

Enrolled in any college 3 

87.5 

88.4 

84.7 

2.8 

2.2 

2 7 *** 

1.1 

0.9 

2.3 

Enrolled in any 2-year college 

51.6 

51.4 

47.0 

4.5 

3.1 

4 4 *** 

1.6 

- 0.1 

3.3 

Enrolled in a California community college 

51.2 

50.8 

46.4 

4.8 

3.1 

4 4 *** 

1.6 

-0.5 

3.3 

Enrolled in any 4-year college 

38.1 

41.7 

42.1 

-4.0 

3.0 

-0.4 

1.6 

3.6 

3.2 

Enrolled in a California State University college 

22.2 

22.3 

22.2 

0.0 

2.6 

0.1 

1.3 

0.0 

2.7 

Enrolled in a University of California college 

9.7 

13.2 

12.7 

-3.0 

2.1 

0.5 

1.1 

3.5 

2.2 


(continued) 



Table 4.1 (continued) 



Average Outcome Levels 

CFC Program 

PBS Program 

PBS Program 


CFC PBS 

vs. Control 

vs. Control 

vs. CFC Program 


Program Program Control 

Standard 

Standard 

Standard 

Outcome (%) 

Group Group Group 

Difference Error 

Difference Error 

Difference Error 


Third term 


Enrolled in any college 3 

80.0 

81.4 

79.0 

1.0 

2.5 

2.4 * 

1.3 

1.4 

2.6 

Enrolled in any 2-year college 

46.2 

43.9 

41.4 

4.8 

3.1 

2.5 

1.6 

-2.3 

3.2 

Enrolled in a California community college 

45.5 

42.9 

40.8 

4.7 

3.1 

2.1 

1.6 

-2.6 

3.2 

Enrolled in any 4-year college 

34.5 

38.5 

38.2 

-3.7 

3.0 

0.3 

1.6 

4.0 

3.2 

Enrolled in a California State University college 

19.7 

20.5 

19.9 

-0.2 

2.5 

0.6 

1.3 

0.8 

2.6 

Enrolled in a University of California college 

9.0 

12.0 

11.7 

-2.7 

2.0 

0.2 

1.0 

3.0 

2.1 

Sample size (total = 4,921) 

279 

1,361 

3,281 








SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

National Student Clearinghouse data were not found for 320 students (6.5 percent of the sample). 
a A small proportion of students were enrolled at more than one institution. 



In earlier MDRC studies of perfonnance-based scholarships, impacts on enrollment 
during the first program term rarely occurred. 6 This is likely because students were recruited to 
participate after they were already registered, or similarly committed to register (such as through 
participation in an installment plan), leaving little to no opportunity for the program to affect 
enrollment in the first term. Given the notification of scholarship eligibility in June for the CFC- 
PBS Program, it is reasonable to expect a similar pattern among students selecting to attend 
four-year institutions, as such institutions typically require students to apply and accept an invi- 
tation to enroll several months prior to the start of the academic term (in other words, before 
June). As a result, the first-term impacts reported above will tend to be concentrated among stu- 
dents attending community colleges or other two-year private institutions, as students can de- 
cide to enroll in these at any time before the add/drop deadline (which may occur as late as Sep- 
tember in such institutions). 

The remaining rows in the panel divide enrollment into the types of institutions where 
students matriculated. As expected, these rows show that all of the impact in first-term enroll- 
ment is being driven by enrollment in two-year colleges, where about half of the students ma- 
triculated. 

Second-Term Persistence 

The second panel of Table 4.1 presents figures on continued enrollment in the second 
tenn, or spring term, after random assignment. The first row of the panel shows that all groups 
of students continued to be enrolled at high rates, but the PBS program group students were 3.7 
percentage points more likely to be enrolled than control group students. There is no statistical 
difference in enrollment for CFC program group students and control group students. Again, the 
impacts are being driven by enrollment in two-year colleges. 

Year-to-Year Persistence 

The last panel shows persistence in the third tenn, or enrollment one full year after ran- 
dom assignment. The first row shows a slight drop in attendance among both program group 
and control group members. About 81 percent of PBS program group members were enrolled, 
compared with 80 percent of CFC program group members and 79 percent of control group 
members. While a drop in enrollment over a year is quite common, PBS program group stu- 
dents were about 2.4 percentage points more likely to return compared with control group stu- 


6 For more information, see Cha and Patel (2010); Miller, Binder, Harris, and Krause (2011); Patel and 
Rudd (2012); and Patel and Valenzuela (2013). 
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dents. 7 In short, the table suggests that the performance-based scholarships resulted in a small 
increase in year-to-year persistence. 

Enrollment Impacts by Scholarship Type 

Table 4.2 shows enrollment outcomes by scholarship type, where the six scholarship 
types are shown across the columns of the table (see Box 4. 1 for how to read a six-way impact 
table). The first entry in the first-term panel shows the average enrollment rate for control group 
members of 84.4 percent, a figure that is identical to that shown in the first panel of Table 4.1. 
The entry in the second column shows no statistically significant difference in outcomes for 
program group members assigned to Scholarship Type 1 (the CFC program group, also shown 
in the first panel of Table 4.1) compared with the control group. The third column shows an in- 
crease in enrollment of 4.1 percentage points for PBS program group students assigned to 
Scholarship Type 2. This impact indicates that 88.5 percent (= 84.4 + 4.1) of PBS program 
group students assigned to the one-term, $1,000 scholarship type matriculated at an institution 
in the fall following random assignment. 

The remaining entries in the first-term panel show the impact associated with enroll- 
ment by a PBS program group student assigned to a particular scholarship type compared with 
the control group mean. The entries are all modest, positive, and statistically significant, which 
suggests that the pooled PBS program group impact reported in the first panel of Table 4. 1 is 
not driven by a particular scholarship type. 8 

The second-term panel shows impacts on enrollment in the spring term. The first col- 
umn replicates the control group enrollment rate of 84.7 percent reported in Table 4.1. The re- 
maining columns show the impact over the control group mean for each scholarship type. The 
table suggests that differences in enrollment impacts can be detected only for Scholarship Type 
3 ($500 in each of two terms) and Scholarship Type 4 ($1,000 in each of two tenns). These two 
scholarship types appear to drive the overall pooled impact reported in Table 4.1. The last panel 
reports on year-to-year persistence, and the entries reveal that only the impact of 4.4 percentage 


7 Drop-off after the summer term is typical and slightly larger on average than reported in the text. Horn 
and Weko (2009), using data from the 2003-2004 cohort of the Beginning Postsecondary Students Longitudi- 
nal Study that are limited to community college students, report that 22.9 percent of students depart in the first 
year, or 77.1 percent were retained (Table 10, p. 33). 

s While the enrollment rates among the various PBS program groups differ from those of the control group, 
they are not statistically different from each other. See Appendix Tables D.l and D.2 for the full matrix of 
comparisons and Box 4. 1 for an explanation of how to read the table. As discussed in Chapter 2, to mitigate 
spurious findings, individual pairs of treatments are examined only when the F-test of whether the coefficients 
for the performance-based scholarship types are jointly equal to zero is significant. See Schochet (2008). 
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Box 4.1 

How to Read the Six-Way Impact Tables in This Report 

Some tables in this report use the format illustrated in the table excerpt below, which displays 
the first-semester enrollment outcome for the six program groups and the control group. The 
first column shows that 84.4 percent of control group students enrolled in the first term. The 
second column shows that students assigned to Scholarship Type 1 (or the one-semester, 
$1,000 scholarship without performance criteria) enrolled at a rate that was 3.5 percentage 
points higher than the control group. The second column also shows the standard error of the 
estimate, 2. 1 percentage points, reported in parentheses below the impact. This difference is 
not statistically significant at conventional levels. 

That is, 3.5 is the difference in the rate of enrollment between the Scholarship Type 1 group 
and the control group, or the estimated average impact of the opportunity to participate in the 
program. The program group for Scholarship Type 1 enrolled at a rate of 87.9 percent, which 
can be calculated by adding the impact of 3.5 percentage points to the outcome of 84.4 per- 
cent. Statistical significance is denoted in the same way as in other tables in this report, and 
the lack of any asterisks indicates that the estimated impact is not significant at the 1 percent, 
5 percent, or 1 0 percent level. 


Outcome (%) 

Control 

Group 

Average 

Impact by Scholarship Type 
1 2 

3 

Performance-based scholarship 
Amount of scholarship per 


No 

Yes 

Yes 

semester ($) 


1,000 

1,000 

500 

Duration of scholarship 


1 semester 

1 semester 2 semesters 

First term 

Enrolled in any college 

84.4 

3.5 

(2.1) 

4.1 * 
(2.2) 

3.9* 

(2.2) 


points for Scholarship Type 5 ($500 in each of four terms) is statistically different from the con- 
trol group mean. 

So what do these findings mean? Overall, the differences in enrollment caused by the 
program are positive, modest in size, and generally statistically significant as compared with 
what would have happened in the absence of the program (the status quo, as represented by the 
control group). 


70 







Increases in matriculation occurred almost unifonnly across all scholarship types and 
appear to be concentrated among students enrolling in two-year institutions (not shown in the 
tables). The uniformity of impacts suggests that modest increases in matriculation can be ac- 
complished with fairly low dollar amounts. In fact, the analyses suggest that scholarships of 
$500 to $1,000, independent of duration, perform equally well in affecting first-tenn enroll- 
ment. 9 These results are among the first to show conclusively that financial aid increases en- 
rollment in postsecondary education and that modest scholarship amounts (typical of private 
scholarships such as those offered by small organizations like local town businesses) can have 
effects on matriculation. 

The findings also suggest that $1,000 performance-based scholarships are equally as ef- 
fective as $1,000 need-based grants in generating matriculation effects (and failing to generate 
persistence effects). Appendix Table D.3 suggests that the difference in impacts between Schol- 
arship Type 1 (the original CFC program scholarship) and Scholarship Type 2 is statistically 
insignificant across each tenn. 10 

The early evidence on the effectiveness of one-year scholarships suggests such scholar- 
ships may affect enrollment while the scholarship is available. For example, Table 4.2 shows 
positive and statistically significant impacts for Scholarship Type 3 ($500 in each of two terms) 
and Scholarship Type 4 ($1,000 in each of two tenns) over the control group mean during both 
the first tenn and second tenn. The impacts appear to fade in the third tenn when program 
group students were no longer eligible. 

In short, perfonnance-based scholarships appear to encourage matriculation and im- 
prove persistence. It is too early to interpret the patterns of enrollment for Scholarship Type 5 
and Scholarship Type 6 students in this report as these scholarships are disbursed over two 
years and a stable pattern may emerge only once the scholarship disbursements are completed. 
As a result of this, it is premature to determine what amounts of aid matter based on the ana- 
lyzed data. 


’Appendix Table D. 1 shows no statistical differences in first-term enrollment between any pair of scholar- 
ship types. Appendix Tables D.4 and D.5 compare $l,000-per-term scholarships with $500-per-term scholar- 
ships and show no meaningful differences. While one could argue that the cross-comparisons shown in Ap- 
pendix Table D.l are insignificant as a result of the sample sizes for each comparison being underpowered, the 
point estimates shown in the table are sufficiently small to be irrelevant in both a policy sense and a practical 
sense. From a policy perspective, differences as small as 1.4 percentage points are not meaningful, as such 
increases are unlikely to move the long-term outcomes (such as graduation and employment) that are of ulti- 
mate concern. In a practical sense, in order to detect differences of the magnitude of 1.8 percentage points with 
80 percent power, a sample size of roughly 5,000 observations would be required in each group. 

10 See Appendix Table D.3 for formal tests of equality of the two scholarship types across the three pro- 
gram terms. 
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The Performance-Based Scholarship Demonstration 
Table 4.2 


Enrollment Impact Over the Control Group, by Scholarship Type: 
First Through Third Terms 


Cash for College Performance-Based Scholarship Study 



Control 

Group 



Impact by Scholarship Type 



Outcome (%) 

Average 

1 

2 

3 

4 

5 

6 

Performance-based scholarship 


No 

Yes 

Yes 

Yes 

Yes 

Yes 

Amount of scholarship per semester ($) 

1,000 

1,000 

500 

1,000 

500 

1,000 

Duration of scholarship 


1 semester 

1 semester 

2 semesters 

2 semesters 

4 semesters 

4 semesters 

First term 

Enrolled in any college 

84.4 

3.5 

(2.1) 

4.1 * 
(2.2) 

3.9 * 
(2.2) 

5.1 ** 
(2.2) 

5.3 ** 
(2.2) 

5 g *** 
(2.2) 

Second term 

Enrolled in any college 

84.7 

2.8 

(2.2) 

3.5 

(2.2) 

5.6 ** 
(2.2) 

3.8 * 
(2.2) 

2.7 

(2.2) 

3.0 

(2.2) 

Third term 

Enrolled in any college 

79.0 

1.0 

(2.5) 

1.8 

(2.5) 

3.2 

(2.5) 

1.9 

(2.5) 

4.4 * 
(2.5) 

0.5 

(2.5) 

Sample size (total = 4,921) 

3,281 

279 

264 

273 

276 

276 

272 


(continued) 



Table 4.2 (continued) 


SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: To analyze whether the performance-based scholarship type predicted registration, a joint test was performed, adjusting for research 
cohort and workshop region. For the first program semester, this test yielded a p-value for the F-statistic of less than 0.01, which is significant at 
the 1 percent level. This suggests that the differences in first-semester registration between scholarship types and the control group are unlikely 
to have occurred by chance. For the second program semester, this test yielded a p-value for the F-statistic of 0.03, which is significant at the 5 
percent level. For the third program semester, this test yielded a p-value for the F-statistic of 0.42, which is not significant at any level. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

Standard errors are reported in parentheses under impact estimates. 

National Student Clearinghouse data were not found for 320 students (6.5 percent of the sample). 





Subgroup Analyses of the Effect of Performance-Based 
Scholarships 

Based on previous research, several subgroups were specified before the analysis began based 
on characteristics before random assignment: gender, race/ethnicity, high school preparedness 
(using high school GPA as a proxy), and motivation for applying for financial aid at baseline. 11 
Table 4.3 shows matriculation impacts for the above subgroups. The first panel shows that 
males in the PBS program group registered at a higher rate (6.9 percentage points higher) than 
males in the control group. 12 The pattern of higher registration is also evident among women. 
However, the difference in first-term enrollment impacts between men and women is not statis- 
tically significant, indicating that perfonnance-based scholarships work equally well for both 
groups. 


The second panel shows matriculation rates by ethnicity. Both Latino and non-Latino 
PBS program group members registered at higher rates than their respective control group coun- 
terparts. Again, the difference in impacts between Latino students and non-Latino students is 
not statistically significant. 

The third panel shows that the program did not affect students who may have been well 
prepared for college, as proxied by having a high school GPA of 3.0 or higher, but it did induce 
less-prepared students to matriculate at higher rates. 13 That is, the program had a differential 
effect on less-prepared students above what could be expected by chance because the difference 
in matriculation impacts between the two groups is statistically significant. 

The last panel of Table 4.3 shows differences by whether students were intrinsically 
motivated to apply for financial aid — that is, whether the factor that motivated them to apply 
for financial aid was internally driven as opposed to their being externally coerced, such as 
through pressure from parents. 14 The table shows that program group students in both sub- 
groups were induced to enroll at greater rates. 

Table 4.4 shows enrollment impacts for the second term by the same subgroups. The 
above patterns are largely repeated, and the program continued to have a differential effect on 


U AB 540 status was also prespecified, blit that subgroup is excluded from all analyses in this report be- 
cause data collected for these students were incomplete as a result of the absence of Social Security numbers 
and poor matches by name and date of birth. See Appendix A. 

12 All analyses in this section focus on the pooled performance-based scholarship program group; the CFC 
program group is omitted from all analyses. 

l3 The sequence of courses taken is an alternative proxy of college preparedness, but that information is 
unavailable for the sample. 

14 Ryan and Deci (2000). 
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The Performance-Based Scholarship Demonstration 
Table 4.3 

Enrollment by Subgroups: First Term 


Cash for College Performance-Based Scholarship Study 




PBS 



Difference 


Sample Program 

Control 

Standard Between 

Subgroup (%) 

Size 

Group 

Group Difference 

Error Subgroups 

Gender 






Male 

1,848 

89.5 

82.6 

^ g *** 

1.8 

Female 

2,793 

89.1 

85.5 

3.6 *** 

1.4 

Total sample size 

4,641 





Latino 






Yes 

2,795 

88.9 

82.9 

6.0 *** 

1.5 

No 

1,806 

89.8 

86.8 

3.0 * 

1.7 

Total sample size 

4,601 





High school preparedness 





ftf 

High school GPA 3.0 or above 

2,222 

93.6 

92.6 

1.0 

1.2 

High school GPA below 3.0 

2,389 

85.3 

76.9 

g 4 *** 

1.8 

Total sample size 

4,611 





Motivation to apply for financial aid 3 






Relative Autonomy Index 0 or above 

3,834 

89.3 

85.0 

4 3 *** 

1.2 

Relative Autonomy Index below 0 

782 

89.5 

81.5 

g 1 *** 

2.9 

Total sample size 

4,616 






SOURCES: MDRC calculations using National Student Clearinghouse data, Exit Survey (Baseline 
Information Form) data, and Cal Grant data provided by the California Student Aid Commission. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance 
levels are indicated as: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

“Motivation to apply for financial aid is defined using the Relative Autonomy Index (RAI), which 
has a range of -18 to 18, where a higher value represents greater autonomous motivation. The RAI is 
calculated as a weighted average: RAI = [Externalx(-2)] + [lntrojected x (-l)] + [Identifiedx(l)] + 
[lntegratedx(2)j. See Deci, Koestner, and Ryan (2001). 
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The Performance-Based Scholarship Demonstration 
Table 4.4 

Enrollment by Subgroups: Second Term 


Cash for College Performance-Based Scholarship Study 




PBS 



Difference 


Sample Program 

Control 

Standard Between 

Subgroup (%) 

Size 

Group 

Group 

Difference 

Error Subgroups 

Gender 






Male 

1,848 

87.4 

83.0 

44 ** 

1.9 

Female 

2,793 

89.1 

85.8 

3.3 ** 

1.4 

Total sample size 

4,641 





Latino 






Yes 

2,795 

88.6 

84.0 

4 6 *** 

1.5 

No 

1,806 

88.2 

86.0 

2.2 

1.7 

Total sample size 

4,601 





High school preparedness 





tf 

High school GPA 3.0 or above 

2,222 

93.1 

92.1 

1.0 

1.2 

High school GPA below 3.0 

2,389 

83.7 

78.0 

5.8 *** 

1.8 

Total sample size 

4,611 





Motivation to apply for financial aid 11 





ft 

Relative Autonomy Index 0 or above 

3,834 

88.2 

85.6 

2.5 ** 

1.2 

Relative Autonomy Index below 0 

782 

90.3 

80.4 

g g *** 

2.9 

Total sample size 

4,616 






SOURCES: MDRC calculations using National Student Clearinghouse data, Exit Survey (Baseline 
Information Form) data, and Cal Grant data provided by the California Student Aid Commission. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance 
levels are indicated as: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

“Motivation to apply for financial aid is defined using the Relative Autonomy Index (RAI), which 
has a range of -18 to 18, where a higher value represents greater autonomous motivation. The RAI is 
calculated as a weighted average: RAI = [Externalx(-2)] + [lntrojected x (-l)] + [Identifiedx(l)] + 
[lntegratedx(2)j. See Deci, Koestner, and Ryan (2001). 
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less-prepared students (proxied by their high school GPA), as the difference in persistence im- 
pacts between the two groups is statistically significant. The exception to the pattern shown in 
Table 4.3 is that a small enrollment impact appears for those students who were not intrinsical- 
ly motivated to apply for financial aid in Table 4.4. That is, while both program groups contin- 
ued to be induced to enroll at higher rates than the control group, the effects were larger for 
those who did not have a strong personal interest in applying for financial aid in their senior 
year of high school. 

Finally, Table 4.5 shows enrollment impacts for the third term (showing year-to-year 
persistence), by the above subgroups. The impacts by gender fade over time, while those for 
Latino students are sustained, although smaller in magnitude. The differential effect for less- 
prepared students disappears for year-to-year persistence. Importantly, the program continued to 
have a large and persistent effect on students who were less intrinsically motivated to apply for 
financial aid at baseline. The table entries suggest that without the CFC-PBS Program, about 75 
percent of such students would be enrolled one year after matriculation. In contrast, the CFC- 
PBS Program induced about 84 percent of students to persist, for an increase of 9 percentage 
points (or a 12 percent increase in enrollment). In short, there is evidence that the CFC-PBS 
Program may work better over multiple semesters for students who may not be intrinsically mo- 
tivated to attend workshops like Cash for College. 


Understanding the Effects of the Scholarship 

Enrollment at California Community Colleges 

The above analyses show that both matriculation and persistence impacts are largely 
driven by enrollment in two-year institutions. The tables also show that the vast majority of 
this enrollment in two-year institutions occurred at California community colleges. To further 
illuminate how the CFC-PBS Program affected program group students, an analysis of aca- 
demic outcomes using data from the California Community College Chancellor’s Office 
(CCCCO) was performed. 15 Table 4.6 shows that PBS program group students registered in 
California community colleges at higher rates than control group students, a finding that was 
reported in Table 4. 1 using Clearinghouse data. There are some differences between Tables 4. 1 
and 4.6 due to a small number of discrepancies between the Clearinghouse and CCCCO data 
provided to MDRC. 


15 Since transcript information is available only for students who enrolled in a community college in Cali- 
fornia, assumptions about the enrollment of other students are needed to perform an intent-to-treat analysis. In 
this section, students not found in the data are assumed not to be enrolled. 
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The Performance-Based Scholarship Demonstration 
Table 4.5 

Enrollment by Subgroups: Third Term 


Cash for College Performance-Based Scholarship Study 




PBS 



Difference 


Sample Program 

Control 


Standard Between 

Subgroup (%) 

Size 

Group 

Group 

Difference 

Error Subgroups 

Gender 






Male 

1,848 

79.2 

76.4 

2.8 

2.1 

Female 

2,793 

82.8 

80.8 

2.0 

1.6 

Total sample size 

4,641 





Latino 






Yes 

2,795 

82.5 

79.1 

3.4 

** 1.6 

No 

1,806 

79.9 

78.9 

1.0 

2.0 

Total sample size 

4,601 





High school preparedness 






High school GPA 3.0 or above 

2,222 

89.5 

88.9 

0.6 

1.4 

High school GPA below 3.0 

2,389 

73.4 

70.1 

3.3 

* 2.0 

Total sample size 

4,611 





Motivation to apply for financial aid 3 





ft 

Relative Autonomy Index 0 or above 

3,834 

81.0 

80.0 

1.0 

1.4 

Relative Autonomy Index below 0 

782 

83.9 

74.9 

9.0 

*** 3 3 

Total sample size 

4,616 






SOURCES: MDRC calculations using National Student Clearinghouse data, Exit Survey (Baseline 
Information Form) data, and Cal Grant data provided by the California Student Aid Commission. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance 
levels are indicated as: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

““Motivation to apply for financial aid is defined using the Relative Autonomy Index (RAI), which 
has a range of -18 to 18, where a higher value represents greater autonomous motivation. The RAI is 
calculated as a weighted average: RAI = [Externalx(-2)] + [Introjectedx(-l)] + [ldentified x (l)] + 
[Integrated x (2)]. See Deci, Koestner, and Ryan (2001). 
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The Performance-Based Scholarship Demonstration 
Table 4.6 

Academic Outcomes in California Community Colleges: 
First Through Second Terms 


Cash for College Performance-Based Scholarship Study 


Outcome 

PBS Program 
Group 

Control 

Group 

Difference 

Standard 

Error 

First term 

Enrolled for any courses (%) 

48.0 

42.8 

5 3 *** 

1.6 

Full time 3 

31.4 

29.0 

2.4 * 

1.4 

Part time b 

12.9 

9.5 

3 5 *** 

1.0 

Average number of credits attempted 

5.7 

5.0 

0 7 *** 

0.2 

College-level credits 

4.7 

4.1 

0.6 *** 

0.2 

Developmental credits 

0.9 

0.9 

0.1 

0.1 

Average number of credits earned 

4.5 

3.9 

0.6 *** 

0.2 

College-level credits 

3.8 

3.3 

0.5 *** 

0.2 

Developmental credits 

0.7 

0.7 

0.0 

0.1 

Number of credits earned (%) 

12 or more 

21.2 

18.4 

2.8 ** 

1.3 

6 to less than 12 

15.8 

13.9 

1.9 * 

1.1 

Term GPA (%) 

3.0 to 4.0 

16.9 

15.0 

1.9 

1.2 

2.0 to 2.9 

15.1 

13.7 

1.4 

1.1 

Less than 2.0 

14.0 

12.1 

2.0 * 

1.1 

No GPA e 

54.0 

59.3 

3 *** 

1.6 

Earned 6 + credits with a term GPA of 2.0 

or greater in fall term (%) 

29.2 

25.8 

3.4 ** 

1.4 

Second term 

Enrolled for any courses (%) 

49.4 

45.7 

37 ** 

1.6 

Full time 3 

31.6 

27.6 

^ Q *** 

1.4 

Part time b 

12.2 

12.3 

- 0.1 

1.1 

Average number of credits attempted 

6.0 

5.3 

0.6 *** 

0.2 

College-level credits 

5.2 

4.7 

0.5 *** 

0.2 

Developmental credits 

0.7 

0.6 

0.1 

0.1 

Average number of credits earned 

4.5 

4.0 

0 4 ** 

0.2 

College-level credits 

4.0 

3.6 

0 4 ** 

0.2 

Developmental credits 

0.5 

0.5 

0.0 

0.1 


(continued) 
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Table 4.6 (continued) 


Outcome 

PBS Program 
Group 

Control 

Group 

Difference 

Standard 

Error 

Number of credits earned (%) 

12 or more 

19.8 

17.1 

2 7** 

1.2 

6 to less than 12 

14.6 

14.2 

0.4 

1.1 

Term GPA (%) 

3.0 to 4.0 

16.7 

15.6 

1.1 

1.2 

2.0 to 2.9 

15.1 

14.2 

0.9 

1.1 

Less than 2.0 

15.7 

13.7 

2.0 * 

1.1 

No GPA e 

52.5 

56.4 

_3 g ** 

1.6 

Earned 6+ credits with a term GPA of 2.0 

or greater in spring term (%) 

25.9 

23.8 

2.1 

1.4 

Earned 6+ credits with a term GPA of 2.0 

or greater in summer term (%) 

3.4 

2.6 

0.8 

0.5 

Cumulative terms 1 through 2 

Average number of credits earned 

8.9 

7.9 

Y o *** 

0.4 

Sample size (total = 4,642) 

1,361 

3,281 




SOURCE: MDRC calculations using California Community College Chancellor’s Office transcript data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 
a Full-time enrollment is defined as 12 or more credits attempted. 
b Part-time enrollment is defined as 6 to 12 credits attempted. 

c The “No GPA” category includes students who did not enroll for any courses at a California 
community college. 


The greater detail provided by the data in Table 4.6 shows that the intervention in- 
creased both full-time and part-time enrollment in the first term. PBS program group students 
attempted and earned more college credits at California community colleges than control group 
students, although the impact is modest (less than one-quarter of a course). Interestingly, the 
gain in credits earned seems to be driven by college-level credits. While the average level of 
credits earned is low, any gain in degree-applicable credits is desirable as such gains may place 
students on a shorter trajectory to completing their degrees. The last row of the panel shows a 
slight increase (3.4 percentage points) in the proportion of PBS program group students who 
met the academic benchmarks of earning six or more credits with a tenn GPA of 2.0 or better. 


80 







The panel for the second program term shows similar patterns: increased enrollment (or 
second-term persistence), a greater number of credits attempted, and a higher number of credits 
earned (concentrated among college-level credits). Overall, PBS program group students accu- 
mulated an average of one credit more than control group students after two terms. 

The detailed CCCCO data also provide more insight into the proportion of students 
who met the scholarship benchmarks but did not collect a payment. In the first term, 44.3 per- 
cent of program group students (603 students) enrolled at least part time at an eligible California 
community college. Of these students, around 9 percent did not receive their enrollment pay- 
ments (not shown in the table). Similarly, 397 students were eligible for the performance award 
in the first term (that is, they were assigned to one of the Scholarship Types 2 through 6 and 
earned six or more credits with a term GPA of 2.0 or better), but around 17 percent of these stu- 
dents did not receive their perfonnance payments (not shown in the table). This does imply that 
there is a subset of students who are meeting the criteria to earn the award but not submitting 
their documentation in order to receive payment. 

How the CFC-PBS Program Interacts with State Financial Aid 

A student’s financial aid package contains all the various forms of aid that a student is 
scheduled to receive in a specific term or academic year. Its composition relies on many differ- 
ent factors, such as the processing of the U.S. Department of Education’s FAFSA, levels of 
state aid generosity, student preferences, and institutional rules. In California, low-income stu- 
dents are eligible for a number of financial aid programs, but the Cal Grant program (described 
in Chapter 1) and the Board of Governors (BOG) Fee Waiver are among the most generous and 
well-known. Indeed, the eligibility criteria for the CFC-PBS Program targeted students who 
were likely to be eligible for at least one of the Cal Grant programs and required all students to 
submit the Cal Grant GPA Verification Form before the deadline. 

In this way, the program was expected to increase the amount of additional financial aid 
provided to students while simultaneously providing an incentive for academic achievement. 
The scholarship amounts of $500 and $1,000 per term were selected to be supplemental to other 
aid. Yet a smaller number of students than expected were awarded Cal Grants (see Table 2.3). 
While virtually all CFC-PBS students applied for a Cal Grant, about 64 percent were awarded 
one. Around 8 percent of CFC-PBS students did not meet the high school GPA thresholds for 
Cal Grant A or Cal Grant B. In addition, around 13 percent of CFC-PBS students had a family 
income level that exceeded the Cal Grant B income limits, but the students failed to meet the 
Cal Grant A GPA requirement. Thus, about 21 percent of CFC-PBS students were not awarded 
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a Cal Grant award for reasons related to GPA and family income requirements. 16 The remaining 
1 5 percent may not have been awarded a Cal Grant due to other factors such as differences in 
Cal Grant processing efficiency by institution type. 17 

A potential implication of students not being awarded all of the aid for which they are 
eligible is an increase in the relative importance of the perfonnance-based scholarship in the 
financial aid package. However, the directionality of this increase on student behavior is un- 
known. That is, the effect of aid may be nonlinear such that a threshold, or foundational, amount 
of aid may induce a change in behavior that differs from that caused by aid that is simply added 
to the threshold amount. 18 Indeed, the various perfonnance-based scholarship programs aim to 
have the scholarships enter the aid package as last-dollar aid, adding to other sources of aid ra- 
ther than serving as the only source. Since it is unknown at what point the marginal effect of 
additional aid rises or falls as aid increases, the presence or absence of threshold aid (such as the 
Cal Grant in the CFC-PBS Program) could affect behavioral patterns. In other words, if CFC- 
PBS students are not awarded aid for which they are otherwise eligible, the impacts may reflect 
the effect of perfonnance-based scholarships in lieu of other aid rather than as a last-dollar 
award, as designed. 

To explore this, students who were eligible for Cal Grant A or B at baseline were com- 
pared with ineligible students, as shown in Table 4.7. The table shows higher levels of matricu- 
lation among students eligible for the two types of Cal Grants at baseline. Specifically, 92 per- 
cent of PBS program group members who were eligible for Cal Grant A and B matriculated, 
compared with 87 percent of control group members. These levels are higher than the 82 per- 
cent of Cal Grant-ineligible PBS program group members who matriculated and the approxi- 
mately 76 percent of Cal Grant-ineligible control group members who matriculated. However, 
the impacts from these two groups are not statistically different, as indicated in the last column 
of the table. 

In contrast to the above, enrollment patterns between two-year and four-year colleges 
differ dramatically between the groups. Slightly more than half of students who were eligible 


16 Given the income threshold requirements for the study, eligibility for Cal Grant A in this analysis is 
proxied solely by having a high school GPA of 3.0 or higher. Cal Grant B eligibility is calculated using de- 
pendency status, household size, and household income from the FAFSA in order to determine whether stu- 
dents are below the Cal Grant B income ceilings, which are lower than those required for Cal Grant A. 

17 For example, one reason for Cal Grant disqualification is that the student does not seem to have enough 
financial need based on the first California college listed on his or her FAFSA. (See the California Student Aid 
Commission website for more information: www.calgrants.org.) If such a student attends his or her lower- 
choice California college and that college does not seek to reconcile its eligibility list with that of the California 
Student Aid Commission (the fiscal agent for Cal Grants), then an otherwise eligible student will not receive an 
award. There are other scenarios that could similarly result in nonaward. 
ls Dynarski (2003); Long (2004). 
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The Performance-Based Scholarship Demonstration 
Table 4.7 


Enrollment Outcomes, by Cal Grant A or B Eligibility Prior to Random Assignment: 

First Through Third Terms 

Cash for College Performance-Based Scholarship Study 

Eligible for Cal Grant A or B Not Eligible for Cal Grant A or B 

PBS PBS Difference 

Program Control Standard Program Control Standard Between 

Outcome (%) Group Group Difference Error Group Group Difference Error Subgroups 


First term 


Enrolled in any college 2 

92.0 

87.4 

4 # 6 *** 

1.2 

82.2 

75.7 

6.4 ** 

3.0 

Enrolled in any 2-year college 

41.6 

38.9 

2.7 

1.8 

68.4 

59.4 

9 0** 

3.5 

Enrolled in a California community college 

41.1 

38.5 

2.6 

1.8 

68.4 

58.8 

9 o *** 

3.5 

Enrolled in any 4-year college 

52.2 

50.5 

1.7 

1.8 

14.1 

16.8 

-2.7 

2.6 

Enrolled in a California State University college 

27.3 

26.0 

1.3 

1.6 

10.4 

11.5 

-1.1 

2.3 

Enrolled in a University of California college 

17.2 

16.6 

0.6 

1.4 

1.1 

1.3 

-0.2 

0.8 

Second term 

Enrolled in any college 2 

91.2 

87.6 

3.6 *** 

1.2 

80.1 

76.7 

3.4 

3.0 

Enrolled in any 2-year college 

46.2 

43.4 

2.8 

1.8 

68.2 

61.6 

6.6 * 

3.5 

Enrolled in a California community college 

45.6 

42.8 

2.8 

1.8 

67.4 

60.8 

6.6 * 

3.5 

Enrolled in any 4-year college 

50.6 

49.6 

1.0 

1.8 

13.8 

16.8 

-3.0 

2.6 

Enrolled in a California State University college 

26.3 

25.3 

1.0 

1.6 

10.0 

11.2 

-1.1 

2.2 

Enrolled in a University of California college 

17.0 

16.3 

0.7 

1.4 

1.1 

1.3 

-0.2 

0.8 


(continued) 



Table 4.7 (continued) 


Eligible for Cal Grant A or B Not Eligible for Cal Grant A or B 

PBS PBS Difference 

Program Control Standard Program Control Standard Between 


Outcome (%) 

Group 

Group Difference 

Error 

Group 

Group Difference 

Error Subgroups 

Third term 









Enrolled in any college 3 

85.4 

83.0 

2.4 * 

1.4 

70.7 

67.0 

3.6 

3.3 

Enrolled in any 2-year college 

39.3 

38.1 

1.2 

1.8 

59.2 

53.7 

5.4 

3.6 

Enrolled in a California community college 

38.3 

37.6 

0.7 

1.8 

57.7 

52.6 

5.1 

3.6 

Enrolled in any 4-year college 

47.3 

45.7 

1.6 

1.8 

11.8 

13.4 

-1.6 

2.4 

Enrolled in a California State University college 

24.9 

23.0 

1.9 

1.6 

7.0 

9.1 

-2.1 

2.0 

Enrolled in a University of California college 

15.4 

15.0 

0.4 

1.3 

1.1 

1.3 

-0.2 

0.8 

Sample size (total = 4,416) 

1,025 

2,503 



273 

615 




SOURCES: MDRC calculations using National Student Clearinghouse data and FAFSA and Cal Grant data provided by the California Student 
Aid Commission. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance levels are indicated as: f f f = 1 percent; 
ff = 5 percent; f = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

MDRC calculated Cal Grant A eligibility using high school GPA (minimum of 3.0); per the study’s eligibility criteria, all students met the Cal 
Grant A/C financial need requirements. MDRC calculated Cal Grant B eligibility using high school GPA (minimum of 2.0), dependency status, 
household size, household income, and the Cal Grant B income ceilings for the 2009-2010 and 2010-2011 academic years. 

National Student Clearinghouse data were not found for 278 students (6.3 percent of the sample). 
a A small proportion of students were enrolled at more than one institution. 



for Cal Grants matriculated at four-year institutions. Further, consistent with the earlier reported 
patterns, impacts on matriculation occurred only at two-year institutions. Students who were 
ineligible for Cal Grant A or B overwhelmingly enrolled at two-year colleges, and the impact 
on enrollment at California community colleges for this group (about 10 percentage points) is 
significantly larger than the impact on enrollment for Cal Grant-eligible students (2.6 percent- 
age points). 

Overall, the pattern of findings in Table 4.7 suggests that the CFC-PBS Program in- 
duced larger enrollment impacts at California community colleges among Cal Grant-ineligible 
students. 

Other Behavioral Changes 

The goal of this section is to try to reveal a bit more about how and why these patterns 
in impacts appear by exploring some of the causal mechanisms depicted in the theory of change 
illustrated in Figure 1.2. Specifically, the analysis explores whether the program affected stress, 
levels of effort, or student employment. 

Stress 

Table 4.8 reports on measures of social support, health, and motivation for the program 
group and control group. The first row of the table shows that there is little difference in receiv- 
ing the educational support of friends between the program group and the control group. Simi- 
larly, both groups seem to have a low prevalence of depression and anxiety, based on K6 scores 
of around 7.0 (scores in excess of 12 are typically considered at high risk of psychological dis- 
tress). 19 In fact, slightly more than 10 percent of students in both groups are at high risk of dis- 
tress by this measure. 

Motivation 

The second panel of Table 4.8 reports on the effect of the CFC-PBS Program on moti- 
vation. The Relative Autonomy Index for completing course work — an overall measure of 
whether motivation to complete course work is internally driven — is fairly low (around a 2.5 
on a scale of - 1 8 to 1 8), indicating relatively low levels of intrinsic motivation. However, the 
levels are similar across the program group and the control group, indicating that the scholarship 
does not meaningfully affect relative autonomy. 

However, that does not mean that the scholarship has no effect on motivation generally. 
Motivation can be divided into two broad types: autonomous (performed out of interest and 


19 Kessler et al. (2002). 
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The Performace-Based Scholarship Demonstration 
Table 4.8 

PBS Survey Responses: Social Support, Health, and Motivation 


Cash for College Performance-Based Scholarship Study 


Outcome 

Sample 

Size 

PBS 

Program 

Group 

Control 

Group 

Standard 
Difference Error 

Social support and health 

Friends value education 11 (average) 

3,688 

1.6 

1.7 

-0.1 ** 0.0 

K6 score for psychological distress’ 3 (average) 

3,686 

7.0 

6.9 

0.1 0.2 

Indicator of high psychological distress 0 (%) 

3,686 

12.6 

11.5 

1.0 1.1 

Motivation 

Motivation to complete course work 1 ' (average) 

Relative Autonomy Index 

3,687 

2.7 

2.6 

0.1 0.1 

External regulation subscale 

3,687 

5.6 

5.6 

0.1 * 0.0 

Introjected regulation subscale 

3,688 

4.0 

3.8 

0.1 ** 0.1 

Identified regulation subscale 

3,687 

6.2 

6.1 

0.1 *** 0.0 

Integrated regulation subscale 

3,687 

5.9 

5.7 

0.1 *** 0.0 

Sample size 

3,689 

1,163 

2,526 



SOURCE: MDRC calculations using responses from the Performance-Based Scholarship 12-Month 
Survey. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Missing values are not included in individual variable distributions. 

a A six-question scale measuring friends’ opinions about college; response categories range from 1 = 
“extremely” to 4 = “not very.” Questions are averaged. 

b A six-question scale measuring nonspecific psychological distress; response categories range from 0 
= “none of the time” to 4 = “all of the time.” Responses are summed across all six questions, and the K6 
score thus ranges from 0 to 24. Kessler et al. (2002). 

c An indicator of the K6 screening scale measure of psychological distress exceeds 12. 
d Motivation to complete course work is defined used the Relative Autonomy Index (RAI), which has 
a range of -18 to 18, where a higher value represents greater autonomous motivation. The RAI is 
calculated as a weighted average: RAI = [Externalx(-2)] + [Introjectedx(-l)] + [Identifiedx(l)] + 
[lntegratedx(2)j. See Deci, Koestner, and Ryan (2001). 
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closely related to intrinsic motivation) and controlled (performed because of perceived external 
pressure and closely related to extrinsic motivation). For example, reading a book for pleasure is 
autonomously or intrinsically motivated, while reading a book only to avoid failing a course is a 
type of controlled motivation (or extrinsic motivation). The CFC-PBS Program seems to affect 
controlled regulation, or the latter type, as indicated by the subscale in Table 4.8. Some may 
consider this a negative outcome since previous research on children suggests that controlled 
motivation results in children not desiring to pursue an activity without a stimulus. 20 However, it 
is not clear whether such an interpretation is warranted for college students and adults, as many 
systems of daily life are configured in this way (such as receiving pay for work performed). 

In fact, the table also shows that autonomous regulation — a proxy for intrinsic, or self- 
induced, motivation — is also slightly higher for the program group than the control group, as 
indicated by the subscale. While the intervention increases motivation in potentially contradic- 
tory ways, these indices suggest that the end result is an overall increase in motivation for 
course work (albeit a very small one). 

Participation in Supplemental Learning Activities 

Table 4.9 shows survey responses concerning students’ participation in supplemental 
learning activities. Most of the table entries are conditional on enrollment and therefore repre- 
sent nonexperimental estimates (hence, shown in italics). As a result, the findings should be re- 
garded as descriptive and not causal. The table shows that about 40 percent of students in the 
program group and 37 percent of control group students attended a course with the intention of 
improving their study skills, and a much smaller number paid for tutoring services. 

The table shows virtually identical patterns of seeking instructor assistance outside of 
class between the program group and the control group. In contrast, there was a small increase 
in the percentage of program group students participating in study groups. 

Other evidence suggests that the CFC-PBS Program resulted in students devoting more 
of their time to educational activities and less time to leisure activities. 21 Specifically, students 
seemed to spend more time preparing for tests in the first term and more time studying or doing 


^'Understanding the type of motivation may matter because some research suggests that extrinsic motiva- 
tion is difficult to maintain after the withdrawal of the externally motivating factor. In fact, the literature on 
incentives is populated with laboratory examples of possible negative consequences of increases in extrinsic 
motivation. However, other research suggests that is unclear whether increasing motivation this way is harmful 
in the long run and whether it is harmful for adults. See Deci, Koestner, and Ryan (2001). This strand of re- 
search suggests that the type of extrinsic motivation matters more than the mere presence of it. See Ryan and 
Connell (1989). 

21 Barrow and Rouse (2013). 
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The Performace-Based Scholarship Demonstration 
Table 4.9 

PBS Survey Responses: Educational Experiences 
Cash for College Performance-Based Scholarship Study 



Sample Program 

Control 

Standard 

Outcome (%) 

Size 

Group 

Group Difference Error 

Currently attending or ever enrolled in 
postsecondary institution beyond high school, 

after random assignment 

3,689 

96.6 

94.9 

1.7 ** 0.7 

Among those currently or ever enrolled in a 
postsecondary institution beyond high school: a 

Took a course to improve study skills 

3,497 

40.0 

37.0 


Paid for tutoring services 

3,497 

3.9 

3.6 


Sought assistance from instructor outside of class: 

Very o ften 

3,514 

4.6 

5.3 


Often 

3,514 

14.0 

14.6 


Sometimes 

3,514 

59.4 

58.7 


Never 

3,514 

21.4 

19.7 


Not applicable 

3,514 

0.6 

1.6 


Used academic services outside of class: 

Very often 

3,497 

8.6 

9.3 


Often 

3,497 

19.6 

20.1 


Sometimes 

3,497 

44.4 

41.1 


Never 

3,497 

26.1 

27.5 


Not applicable 

3,497 

1.3 

1.9 


Participated in study group with other students: 

Very o ften 

3,515 

7.8 

6.5 


Often 

3,515 

16.1 

18.5 


Sometimes 

3,515 

48.4 

42.2 


Never 

3,515 

25.5 

30.1 


Not applicable 

3,515 

2.1 

2.6 


Sought assistance in choosing classes: 

Very o ften 

3,515 

13.4 

11.5 


Often 

3,515 

24.0 

24.1 


Sometimes 

3,515 

42.7 

42.3 


Never 

3,515 

18.7 

20.2 


Not applicable 

3,515 

1.2 

1.9 


Sample size 

3,689 

1,163 

2,526 



(continued) 







Table 4.9 (continued) 

SOURCE: MDRC calculations using responses from the Performance-Based Scholarship 12-Month 
Survey. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Characteristics shown in italics are calculated for a proportion of the survey respondent sample and 
indicate nonexperimental data. 

Missing values are not included in individual variable distributions. 

Distributions may not sum to 100 percent because of rounding. 

Responses given concern a student’s current or most recent school. 


homework in the second term. Finally, CFC-PBS students seemed to perform better than 
their control group counterparts on measures of learning strategies that are likely to help 
students perform better in class. 

Employment 

Table 4.10 shows employment outcomes for the program group and control group. 
About half of the sample members in each group had worked since random assignment, alt- 
hough PBS program group members were close to 5 percentage points less likely to work than 
control group members. However, about 40 percent of sample members in each group were 
working at the time of the survey fielding nine months later. 


Conclusion 

With any incentive program, there is a concern that students could respond to the incen- 
tives in ways that reasonably reflect the incentive structure but do not necessarily meet desirable 
outcomes from a policy perspective. Some unintended outcomes that could occur with the CFC- 
PBS Program are a reduction in credit hours through fewer credits attempted or an increased 
number of course withdrawals (which may increase the probability of meeting the benchmark 
but possibly delay degree attainment); the selection of easier courses or majors; a reduction in 
work effort for those able to perform above the academic benchmark in the absence of the in- 
tervention (resulting in program group members earning lower grades that cluster around the 
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The Performace-Based Scholarship Demonstration 
Table 4.10 

PBS Survey Responses: Employment 


Cash for College Performance-Based Scholarship Study 


Outcome 

Sample 

Size 

PBS 

Program 

Group 

Control 

Group 

Difference 

Standard 

Error 

Had any jobs since random 

assignment (%) 

3,687 

52.1 

56.8 

-4.7 

*** y 7 

Number of jobs since random 

assignment 

3,687 

0.7 

0.7 

-0.1 

** 0.0 

Currently working for pay 

or profit (%) 

3,687 

39.4 

40.9 

-1.4 

1.7 

Worked in the last 7 days (%) 
Number of jobs in the last 

3,686 

33.7 

35.5 

-1.8 

1.7 

7 days 

Number of hours worked in the last 

3,686 

0.4 

0.4 

0.0 

0.0 

7 days 

3,682 

5.7 

6.2 

-0.5 

0.4 

Sample size 

3,689 

1,163 

2,526 




SOURCE: MDRC calculations using responses from the Performance-Based Scholarship 12-Month 
Survey. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Missing values are not included in individual variable distributions. 


benchmark); and an increase in work effort in counterproductive ways (such as cheating, peti- 
tioning for grade changes, delaying the taking of remedial courses, and so on ). 22 While it is not 


22 Many of these unintended consequences have been reported in the literature on incentives. For example, 
Jacob and Levitt (2003) report increases in teacher cheating in response to high-stakes testing, and Cornwell, 
Lee, and Mustard (2005) report reduced full-time enrollment, increased course withdrawals, and concentration 
among “easier” majors in response to Georgia’s HOPE program. Outside of postsecondary education, numer- 
ous negative income tax experiments — which provide income supplements to public assistance payments to 
“make work pay” — have found that such incentives increase work for some and reduce work hours for others, 
often resulting in a net reduction in work hours for eligible families; see Robins (1985) for a summary. There is 
a large literature in psychology that suggests external rewards can reduce intrinsic motivation such that once 
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possible to effectively rule out all of these scenarios, the data analyzed in this chapter largely 
suggest that the CFC-PBS Program induced students to respond in desired ways: by attending 
college at higher rates, attempting and earning more credits (at least among the sample for 
which there are transcript data), and exhibiting greater effort in academics. 

While it is too early to claim that the CFC-PBS Program is a success based on the data 
period in this report, the analysis so far has produced a few notable findings. Specifically, the 
California program generated modest impacts on enrollment over the first three terms, and these 
impacts are concentrated among students attending two-year colleges. The findings extend 
across a number of at-risk subgroups. There is some evidence that the program had larger ef- 
fects on enrollment for students with high school GPAs below 3.0 and students who were not 
intrinsically motivated to apply for financial aid at baseline. These groups may proxy students 
who are less likely to attend college, all else being equal. However, it is too early to draw con- 
clusions on what amounts of scholarships are most effective because the complete data for 
Scholarship Type 5 ($2,000 over two years) and Scholarship Type 6 ($4,000 over two years) 
are not yet available. 


the rewards are removed, productivity declines. See Eisenberger and Cameron (1996) for a meta-analytic re- 
view that suggests the limited conditions under which these negative outcomes occur. 
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Chapter 5 


Cost Analysis of the Cash for College 
Performance-Based Scholarship Program 


The analysis in this chapter provides an early examination of the cost of the Cash for College 
Performance-Based Scholarship (CFC-PBS) Program. Since payments for Scholarship Type 5 
($2,000 over two years) and Scholarship Type 6 ($4,000 over two years) are spread across four 
terms, extending beyond the available data covered in this report, the analysis is limited to 
Scholarship Types 1 through 4. (See Box 5.1 for brief descriptions of the scholarship types 
included in the analysis.) Cost estimates will be updated in future Performance-Based Scholar- 
ship (PBS) Demonstration analyses. As a result of the current follow-up limitations, the analy- 
sis does not comment on the program’s cost-effectiveness or draw conclusions as to whether 
the program’s impacts justify its costs. Instead, the chapter examines the following research 
questions: 

• How much did each scholarship cost? Specifically, how much was spent on 
scholarship payments and how much was spent on program administration? 

• What percentage of scholarships offered to students were actually paid? Spe- 
cifically, how did this vary across the different scholarship types and how did 
this vary based on payment characteristics? 


Key Findings 

• Scholarship payments made up the majority of program costs. Across 
the four scholarship types covered in this analysis, between 61 percent and 
77 percent of CFC-PBS Program costs were paid directly to students in the 
form of scholarships. The remaining 23 percent to 39 percent of program 
costs paid for program administration. 

• The cost to administer scholarships increases as more performance re- 
quirements are added. Administrative costs varied from $247 to $359 per 
program group member. Scholarship Type 1, with no performance require- 
ments, had the lowest administrative cost, while Scholarship Types 3 and 4, 
with the most rigorous performance requirements, had the highest. 

• Students received a lower proportion of scholarships offered when pay- 
ments were based on academic performance and spread over a longer 
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Box 5.1 


Scholarship Types Analyzed in Chapter 5 

Chapter 5 analyzes the cost of the CFC-PBS Program. Because cost data are limited to the 
period of three program semesters, the analysis in the chapter is limited to Scholarship 
Types 1 through 4. 

Enrollment payments are contingent upon a student enrolling at least half time in an ac- 
credited college or university. Performance payments are contingent upon a student main- 
taining a “C” or better grade point average and earning at least six credits. 

Scholarship Types 1 through 4 have the following characteristics: 

• Scholarship Type 1: This scholarship is not performance-based. Students can receive 
$1,000 as a lump sum upon verifying their enrollment in an accredited college or uni- 
versity in the fall after high school graduation. 

• Scholarship Type 2: Students can receive up to $1,000 in two payments over the 
course of one semester. The first payment is contingent on enrollment, and the second 
is contingent on performance. 

• Scholarship Type 3: Students can receive up to $1,000 in three payments over the 
course of two semesters. In the first semester, students can receive $250 for enrollment 
and $250 for performance. In the second semester, students can receive $500 for per- 
formance. 

• Scholarship Type 4: Students can receive up to $2,000 in three payments over the 
course of two semesters. In the first semester, students can receive $500 for enrollment 
and $500 for performance. In the second semester, students can receive $1,000 for per- 
formance. 


period of time than when payment was based on enrollment in the first 
semester. Across the four scholarship types analyzed, payments based on en- 
rollment paid at least 80 percent of the amount offered. Payments based on 
academic perfonnance in the first semester paid 55 percent to 62 percent of 
the amounts offered. Payments based on academic perfonnance in the sec- 
ond semester paid only 40 percent to 44 percent of the amounts offered. 

• All else being equal, scholarships with more performance requirements 
cost less than scholarships with fewer performance requirements. For 

example, Scholarship Types 1, 2, and 3 all offered $1,000 scholarships, but 
their costs ranged from $920 to $1,042. (These costs include scholarship 
payments and the cost of administration.) These cost differences are a prod- 
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uct of the unique payment requirements of each scholarship. For example, 
Scholarship Type 1 had the highest cost because it was paid based on en- 
rollment in the first semester and did not include performance requirements. 
In comparison, Scholarship Type 3 had the lowest cost because fewer stu- 
dents received the performance payment. 


Methodology 

This chapter estimates the cost of the CFC-PBS Program in California. Cost estimates are based 
on actual scholarship payments and program expenditures over two-and-a-half years (from Jan- 
uary 2009 to June 201 1). 1 The CFC-PBS Program had six types of scholarships. However, this 
analysis estimates only the costs of Scholarship Types 1 through 4. 

The analysis aims to exclude costs that are not part of the steady-state operation of the 
program, as it is believed that this provides the most infonnative estimate for policymakers, 
foundations, colleges, or others who may be interested in implementing such a program. As a 
result, all costs related to MDRC’s evaluation have been excluded, as have start-up costs. 2 Ad- 
ditionally, all costs have been categorized as either scholarship payments or program admin- 
istration costs. These categories are estimated using slightly different methodologies. The cost 
of scholarship payments is estimated using data provided by the California Student Aid Com- 
mission. These costs are tracked at the student level and can therefore be easily assigned to 
scholarship types, which allows for a precise estimate of how much was paid in scholarships for 
each scholarship type. The cost of program administration is estimated using program expendi- 
ture data as recorded by the Los Angeles Area Chamber of Commerce (L.A. Chamber). This 
infomiation was tracked at the program level (that is, it was not tracked by scholarship type). 
Therefore, in order to estimate the cost of administration for each scholarship type, a number of 


'The first cohort did not start until fall 2009, but expenses are considered as starting in January 2009 be- 
cause a number of expenses related to the first cohort were incurred in the first two quarters of 2009. These 
expenses would occur each time a new cohort of students began the program and include the cost of staff time 
to recruit and enroll students in the program. 

2 For example, this analysis excludes costs related to the design and development of the PBS Scholars 
website but includes costs related to its maintenance over the analysis period. Start-up costs can be expected to 
vary widely from setting to setting, depending on the existing infrastructure and resources available. For exam- 
ple, the CFC-PBS Program was built on the preexisting payment infrastructure of the Cash for College pro- 
gram. If a similar program were implemented in a different setting, a payment system might have to be created 
wholesale, suggesting a much higher initial level of resource use. Alternatively, if a similar program were im- 
plemented by an institution that already had a robust tool for web-based communication with students, much 
less effort would need to be devoted to developing a website analogous to the PBS Scholars website. In each of 
these scenarios, start-up costs would be likely to be very different from those of the CFC-PBS Program. In 
contrast, the steady-state costs of operating the programs could be expected to be relatively comparable to the 
ongoing costs of the CFC-PBS Program. 
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marginal cost estimates were developed. 3 See Box 5.2 for a discussion of how marginal cost 
assumptions influence the estimates of the costs of individual scholarship types. In addition, 
Appendix E provides further details about the marginal cost methodology used in this analysis. 


Cost Analysis 

The structure of the CFC-PBS Program was different from the structure of other PBS Demon- 
stration sites across the country. Specifically, the CFC-PBS Program was run by a central office 
at the state level and placed the burden of proof regarding perfonnance on students. These de- 
sign elements likely helped to control administrative costs. 

Cost per Program Group Member 

Figure 5.1 shows the cost per program group member by scholarship type, over the life 
of the scholarship. Overall, the program’s total cost ranged from $920 (Scholarship Type 3 — 
$500 per tenn over two terms) to $1,537 (Scholarship Type 4 — $2,000 in three payments over 
two semesters). The largest component of cost for all four scholarship types was scholarship 
payments to students (ranging from a low of 61 percent for Scholarship Type 3 to a high of 77 
percent for Scholarship Type 4). 4 The remaining 23 to 39 percent of the total spending was as- 
sociated with program administration (on average, 28 percent). Scholarship Type 1, with no 
perfonnance requirements, had the lowest administrative cost, at $247, while Scholarship Types 
3 and 4, with the most rigorous perfonnance requirements, had the highest, at $359. 5 

Though not shown in Figure 5.1, the most costly component of program administration 
was personnel, accounting for between 41 percent and 46 percent of administrative spending 
across the four scholarship types analyzed. It is worth noting that, as discussed in Chapter 3, 
Cash for College and L.A. Chamber staff members incorporated CFC-PBS-related work into 
their existing responsibilities, so this estimate represents the costs associated with the time they 
shifted to CFC-PBS as well as some spending on independent contractors, rather than the cost 
of bringing on additional people. Nonpersonnel spending related to student tracking represents 
the next most costly component of administrative expense, making up 20 percent to 22 percent 


3 The marginal cost assumptions are described in detail in Appendix Table E.l. 

4 The cost analysis uses data on actual payments that were made to students. It is worth noting that there 
may have been some students who met enrollment or performance benchmarks but did not receive payments 
because they did not submit the necessary verification. 

5 Because administrative costs were tracked for the program overall, rather than by scholarship type, the 
administrative cost for each scholarship type was estimated by creating a marginal cost per unit for each com- 
ponent of the program’s administrative costs. See Appendix Table E.l for the specific marginal cost assump- 
tions used. 
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Box 5.2 


Marginal Cost Methodology 

This text box uses a hypothetical example to illustrate how marginal cost assumptions influ- 
ence the estimated cost to administer a scholarship. Imagine that it costs $100 to administer two 
scholarships, Scholarship A and Scholarship B. The specific cost per scholarship is unknown, 
but information is available about the characteristics of both scholarships. Marginal cost as- 
sumptions based on what is known about each scholarship allow for a reasonable estimation of 
the cost of administration for both scholarships. The examples below show the different types 
of marginal cost assumptions that are used in this analysis. In these examples, the cost of 
Scholarship A and B will change, but the cost to administer the two scholarships (A and B) will 
always be $100. 

• Marginal cost per scholarship offered: Assumes that the primary driver of cost is the 
number of scholarships offered. If Scholarship A is offered to one student and Scholarship 
B is offered to one student, then the marginal cost per scholarship offered would be $50 
($100 total cost divided by two scholarships). Since Scholarship A offered one scholarship, 
the cost of Scholarship A would be estimated at $50 (marginal cost multiplied by the num- 
ber of units, or $50 x 1). Similarly, since Scholarship B also offered one scholarship, the 
cost of Scholarship B is also $50 ($50 x 1). 

• Marginal cost per scholarship semester: Assumes that the primary driver of cost is the 
number of scholarship semesters. If Scholarship A lasts two semesters while Scholarship B 
lasts only one semester, then the marginal cost per scholarship semester would be approx- 
imately $33 ($100 total cost divided by three scholarship semesters). As a result, the cost 
of Scholarship A would be $66 ($33 x 2) and the cost of Scholarship B would be $33 ($33 

xl). 

• Marginal cost per potential payment: Assumes that the primary driver of cost is the 
number of potential payments. If Scholarship A has three potential payments and Scholar- 
ship B has one potential payment, then the marginal cost per potential payment would be 
$25 ($100 total cost divided by four potential payments). As a result, the cost of Scholar- 
ship A would be $75 ($25 x 3) and the cost of Scholarship B would be $25 ($25 x 1). 

These examples illustrate that assumptions about marginal costs can cause the cost of Scholar- 
ship A to change relative to the cost of Scholarship B. For example, if the number of scholar- 
ships offered is the key driver of cost, then Scholarships A and B have the same estimated cost 
($50). On the other hand, if the number of potential payments is considered the key driver, 
Scholarship A would be estimated to cost three times more than Scholarship B ($75 compared 
with $25). 

In the analysis of CFC-PBS administrative costs, some spending categories were best estimated 
using one of the above marginal cost types. For example, training expenses are believed to be 
driven by the number of scholarships offered. Flowever, some administrative cost components 
seem to be driven by more than one factor. For example, personnel costs seem related to the 
number of scholarships offered, the number of scholarship semesters, and the number of poten- 
tial payments. Therefore, a third of the amount spent on personnel is determined by the number 
of scholarships offered, another third is determined by the number of scholarship semesters, 
and the last third is determined by the number of potential payments. Additional details about 
the marginal cost assumptions used in this chapter are available in Appendix E. 
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The Performance-Based Scholarship Demonstration 
Figure 5.1 

Cost per Program Group Member, 
by Scholarship Type, Over the Life of the Scholarship 

Cash for College Performance-Based Scholarship Study 
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Type 4 


SOURCES: MDRC calculations from expenditure data provided to MDRC by the Los Angeles 
Chamber Foundation and scholarship payment data provided by the California Student Aid 
Commission. 

NOTES: Program costs are based on a steady state of operation that excludes research and start-up 
costs. 

Rounding may cause slight discrepancies in sums and differences. 

The expenditure information used to create this table covers the period of January 2009 through 
June 2011. 

To estimate the cost of the components of program administration by scholarship type, a 
marginal cost per unit was created for each component. This marginal cost per unit is calculated 
based on three factors: the number of scholarships offered, the number of scholarship semesters 
(duration), and the number of potential payments. Marginal costs are allocated in a way that treats 
all schools as semester schools. See Appendix Table E.l for the specific marginal cost assumptions 
used in this analysis. 
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of the total. Program administration also paid for communication with and outreach to students, 
including the maintenance of the PBS Scholars website (16 percent to 19 percent of administra- 
tive spending) 6 and support for regional Cash for College workshops (15 percent to 22 percent 
of administrative spending). 7 

At 28 percent on average, the portion of the CFC-PBS Program’s total cost devoted to 
administration seems high compared with that of federal student aid programs. For example, in 
2010 only about 0.1 percent of the federal Pell Grant program’s expenditures was spent on Ad- 
ministrative Cost Allowances (ACAs) disbursed to schools. ACAs only partially offset institu- 
tions’ spending on administration, but it seems unlikely that additional spending by schools 
would cause the ratio to increase much since the base of Pell Grants is so large (a total of over 
$32 billion was awarded in 2010). 8 Meanwhile, in the 2010-2011 award year, just less than 1 
percent of the Federal Supplemental Educational Opportunity Grant program’s cost went to 
ACAs, while the Federal Perkins Loan program spent just 5 percent of its total expenditures on 
ACAs. 9 However, because the CFC-PBS Program is much newer and smaller in scale than 
these long-established federal grant and loan programs, this may not be a fair comparison. Mi- 
cro finance institutions (MFIs) may be a more appropriate comparison. Box 5.3 uses a compari- 
son with MFIs to discuss the factors that might be influencing the administrative cost of the 
CFC-PBS Program. 

Cost of Scholarships Paid and Offered, by Scholarship Type 

Figure 5.2 shows the average amount of scholarship dollars paid per student as a pro- 
portion of the total dollars the student was offered, by scholarship type. Students eligible for 
Scholarship Type 1 received the highest proportion of scholarships offered (80 percent). Stu- 
dents eligible for Scholarship Type 2 received a lower proportion of scholarships offered (67 
percent). Students eligible for Scholarship Types 3 and 4 received the lowest proportion of 
scholarships offered (56 percent and 59 percent, respectively). 


6 While the PBS Scholars website was largely underused, it is included in the analysis because it is a part of 
the intervention that program group members received. However, it is worth noting that it represents an ex- 
penditure that would likely not be repeated if the program were replicated. If costs associated with the website 
were excluded from the analysis, the administrative cost per program group member would decrease to be- 
tween $226 (Scholarship Type 1) and $317 (Scholarship Types 3 and 4). 

7 As Chapter 3 notes, for the purposes of the study, Cash for Colleges workshops were the vehicle to re- 
cruit the study sample and were not a direct part of the intervention being evaluated. However, even if the pro- 
gram had not been part of an evaluation, it would have been necessary to have a strategy to recruit participants, 
so the workshops have been included in this analysis as a program cost. 

8 U.S. Department of Education (2011). 

9 U.S. Department of Education (2012). 
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Box 5.3 


Factors Influencing the Administrative Cost of 
Performance-Based Scholarship Programs 

On average, approximately 28 percent of the total cost of the CFC-PBS Program went to ad- 
ministration. A comparison with microfmance institutions (MFIs), which provide small-scale 
financial services to otherwise underserved entrepreneurs and small businesses, suggests three 
factors that might influence the administrative costs of the CFC-PBS Program: (1) the average 
scholarship amount; (2) the total number of recipients, or scale; and (3) the length of time the 
program has been in operation. A 2009 study that looked at a sample of 555 sustainable MFIs 
found that the median ratio of these institutions’ administrative costs to gross loan portfolio was 
just over 1 1 percent. There was substantial variation in this ratio among the MFIs in the sam- 
ple, which the authors attributed to the factors noted above. In particular, the following points 
are noted: 

• Relative to their value, small loans cost more to administer than larger ones do. A similar 
dynamic is apparent in the CFC-PBS Program: 39 percent of expenditures associated with 
Scholarship Type 3, which offered $1,000 at three payment points over two semesters, 
were administrative, compared with 23 percent for Scholarship Type 4, which offered 
$2,000 over the same time frame. 

• The scale of an MFFs client base appears to increase its loan-to-administrative cost ratio up 
until the MF1 reaches about 2,000 clients, after which additional clients no longer seem to 
improve efficiency. The authors attribute this phenomenon to the labor-intensiveness of 
microcredit. The CFC-PBS Program was similarly labor-intensive, with personnel costs 
representing the most costly component of administrative spending. 

• Finally, the efficiency of MFIs seemed to increase with their age, particularly for the first 
six years in operation, suggesting a learning curve that may be relevant to a performance- 
based scholarship program as well. 

Though this analysis cannot draw any conclusions as to the effects of these factors, it is inter- 
esting to consider how the program’s administrative costs might change in relation to these fac- 
tors based on the comparison with MFIs. Assuming that the CFC-PBS Program responded to 
these factors in a manner similar to MFIs, imagine a hypothetical scenario in which (1) scholar- 
ship amounts were increased threefold (to be more comparable to the Pell Grant average of 
$3,865); (2) the number of students participating increased past the 2,000 threshold, lowering 
the administrative cost per participant by approximately 1 5 percent, which is comparable to ob- 
served MF1 changes; and (3) the program was in operation long enough to lower administrative 
costs by 15 percent. In this scenario, the percentage of the program’s costs associated with ad- 
ministration could be expected to drop from the current proportion of 28 percent to approxi- 
mately 8 percent on average. 


NOTES: The hypothetical example assumes that the Scholars website is no longer a component of the 
program. 

Rosenberg, Gonzalez, and Narain (2009). MFIs are defined as sustainable if their return on their as- 
sets was positive during the study period. 
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The Performance-Based Scholarship Demonstration 
Figure 5.2 

Cost of Scholarships Paid and Offered, 
by Scholarship Type, Over the Life of the Scholarship 

Cash for College Performance-Based Scholarship Study 



Scholarship Type 

SOURCES: MDRC calculations from expenditure data provided to MDRC by the Los Angeles 
Chamber Foundation and scholarship payment data provided by the California Student Aid 
Commission. 

NOTES: Program costs are based on a steady state of operation that excludes research and start-up 
costs. 

The expenditure information used to create this table covers the period of January 2009 through 
June 2011. 
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Cost of Scholarships Paid and Offered, by Payment Type 

Figure 5.3 shows the proportion of scholarship dollars paid to scholarship dollars of- 
fered, by payment type. While all scholarship types included enrollment payments at the be- 
ginning of the first term, only Scholarship Types 2, 3, and 4 included performance payments, 
and only Scholarship Types 3 and 4 had performance payments in the second term. Figure 5.3 
shows that across all four scholarship types, program group members received between 80 
percent and 84 percent of the enrollment payments they were offered in the first term. 10 Pro- 
gram group members who were eligible for Scholarship Types 2, 3, and 4 received a much 
smaller proportion of the performance-based payments they were offered: between 55 percent 
and 62 percent of the performance payments offered in the first term. This number decreased 
even further in the second term, with students eligible for Scholarship Types 3 and 4 receiving 
40 percent and 44 percent, respectively, of the amounts they had been offered as performance 
payments. 11 

Figure 5.3 shows that students tended to receive a smaller proportion of the perfor- 
mance payments they were offered compared with enrollment payments, and that students were 
likely to collect successively fewer scholarship dollars as more payment points were added and 
as those payment points were spread across longer time periods. Therefore, scholarship types 
that require students to meet performance benchmarks at a series of payment points over longer 
periods of time can be expected to pay out fewer scholarship dollars. However, because of the 
expenses associated with processing each scholarship payment, administrative expenses can be 
expected to increase as the number of payment points increases and as they are spread over 
longer time periods. 

This competing dynamic is illustrated by a comparison of the costs of the three scholar- 
ships that offered $1,000 (Scholarship Types 1, 2, and 3). Each type had a different structure, 
which resulted in a different cost. For example, Scholarship Type 1 paid the entire $1,000 based 
on enrollment in the first semester. It had the highest total cost ($1,042), including the highest 
amount paid to students ($796); it also had the lowest cost of administration, at $247. Scholar- 
ship Type 2 spread the $1,000 across two payments: one for enrollment in the first semester and 
another for academic perfonnance in the first semester. As a result, it had a lower total cost 


10 As Box 5.1 illustrates, program group members in the CFC-PBS study were eligible for two different 
types of scholarship payments: enrollment payments and performance payments. Enrollment payments were 
contingent on registration for at least six credits at the beginning of the term, and performance payments were 
contingent on meeting certain academic performance benchmarks. 

1 1 As noted above, there may have been some students who met enrollment or performance benchmarks 
but did not receive a payment as a result of not submitting the necessary verification. It may be reasonable to 
expect that the portion of students who met benchmarks but did not submit verification increased over time. 
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The Performance-Based Scholarship Demonstration 
Figure 5.3 

Cost of Scholarships Paid and Offered, by Payment Type, Over the Life of the Scholarship 
Cash for College Performance-Based Scholarship Study 



Enrollment payment, Performance payment, Performance payment, 

first program term first program term 3 second program term 

SOURCE: MDRC calculations from scholarship payment data provided by the California Student Aid Commission. 

NOTES: Rounding may cause slight discrepancies in sums and differences. NA = not applicable. 

The value of each payment offered may have varied slightly across the three payments due to differences in scholarship offers 
between semester and quarter schools. 

Students assigned to Scholarship Type 1 are eligible for an enrollment scholarship payment only in the first program term, so 
they are excluded from the performance payment in the first term and all measures in other terms. Students assigned to Scholarship 
Type 2 are eligible for scholarship payments only in the first program term, so they are excluded from all measures from the 
second program term onward. 

“Includes performance payments made at quarter institutions during the winter term, which represent a small portion of the 
sample. 


($951) because it paid less to students ($672), even though it paid slightly more for administra- 
tion ($278). Scholarship Type 3 spread the $1,000 across three payments: one for enrollment in 
the first semester, one for academic performance in the first semester, and one for academic per- 
formance in the second semester. As a result, it had the lowest total cost ($920) because it paid 
the least to students ($560), even though it spent the most on program administration ($359). 
Overall, it appears that despite increased administrative costs, scholarships with more perfor- 
mance requirements cost less than scholarships with fewer performance requirements. This 
means that fewer students receive funding when scholarships are based on perfomrance than 
when they are based on enrollment. The possible implications of this reduction in funding are 
discussed in greater detail in Chapter 6. 


Conclusion 

The early analysis of Scholarship Types 1 through 4 indicates that the CFC-PBS Program cost 
between $920 and $1,537 per student. Scholarship payments accounted for the majority of pro- 
gram costs, while administrative costs made up between 23 percent and 39 percent of program 
expenditures. Administrative costs varied across the scholarship types and increased as perfor- 
mance requirements became more stringent. Meanwhile, on the whole, students received only a 
portion of the scholarship amount they were offered. Students in the CFC-PBS Program tended 
to collect a smaller portion of the performance payments they were offered compared with en- 
rollment payments, and they received successively less money the more payment points a 
scholarship had and the greater period of time the payment points were spread across. As a re- 
sult, when all else is equal, scholarships with more perfomrance requirements cost less overall 
than scholarships with fewer requirements. 
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Chapter 6 

Policy Implications and Conclusions 


This report discusses findings from a unique experimental study of scholarships issued through 
a statewide program. The Cash for College program, an initiative sponsored by the California 
Student Aid Commission (the agency responsible for administering the state’s financial aid 
programs), provided distinctive features conducive to a large-scale experimental study. First, the 
goal of Cash for College — to inform California high school seniors and their parents about 
financial aid opportunities and to help them complete the federal Free Application for Federal 
Student Aid (FAFSA) and state Cal Grant application — enabled the evaluation of the effect of 
additional financial aid for a very large group of students. Second, the concentration of Cash for 
College workshops in schools and communities that are low-income or have low college-going 
rates (or both) ensured that scholarships were aimed toward students at the highest risk of not 
matriculating. Finally, Cash for College’s practice of randomly selecting at least one attendee at 
every workshop who completes the FAFSA to receive a $1,000 scholarship (which is not 
performance-based), as a way to boost workshop attendance, provided a platform for offering 
additional scholarships (which are performance-based), to which students would be similarly 
randomly assigned. 

While the strong Cash for College partnership may be distinctive, similar approaches 
may be possible for state agencies or large and private scholarship providers. In this way, the 
results of the study speak both to the ability to implement such a program on a large scale and to 
the efficacy of the strategy in helping students persist and be academically successful in college. 

Overall, this report finds the following: 

• The Cash for College Performance-Based Scholarship (CFC-PBS) 
Program was largely implemented as designed. 

Despite the complexity of the evaluation and potentially large burden on students, there 
were very few instances of erroneous disbursements, and students largely submitted documents 
as required. 

• The CFC-PBS Program generates a modest, positive impact on first- 
term enrollment of about 5 percentage points above the control group 
average enrollment rate of 84.4 percent. 

This effect extends to numerous subgroups, such as males, females, and students of 
Hispanic/Latino ethnicity. The program has stronger effects for those students with lower high 
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school grade point averages (GPAs) and those students who may not be intrinsically motivated 
to apply for financial aid. 

• Positive impacts of the program are concentrated among students 
attending two-year colleges (specifically, California community colleges). 

This finding is reasonable given the timing of the notification of scholarship eligibility 
in June (during the summer before fall matriculation, after the application deadline for most 
four-year institutions has passed). 

• Scholarship payments made up the majority of program costs. 

Across the four scholarship types covered in this analysis, between 61 percent and 77 
percent of CFC-PBS Program costs were paid directly to students in the form of scholarships. 
As a result of the conditional nature of performance-based scholarships, the cost of scholarship 
payments is noticeably lower than the dollar amount of the scholarships offered to students. All 
else being equal, the cost of scholarship payments decreases as more perfonnance requirements 
are added. 

The remainder of this chapter places these findings in the context of those from other 
studies in the Performance-Based Scholarship (PBS) Demonstration and research in the 
literature on incentives in postsecondary education. It then presents some implications of the 
findings for national policy and for the innovation of financial aid in California specifically. 


Placing the Findings in the Context of the Research on Incentives 

The CFC-PBS Program clearly affected matriculation rates, and those positive effects are 
concentrated among students attending California community colleges. The early findings also 
suggest small but positive effects on persistence in the second and third terms. Given that the 
CFC-PBS Program operates very much like an external (to the institution) scholarship program, 
it is interesting to see how the findings compare with others in the literature, as such a 
comparison can provide a context for understanding the magnitude of the effects of the CFC- 
PBS Program. 

Over the last 12 years, there have been at least nine randomized controlled trials of 
incentive-based grants (including the studies in the PBS Demonstration), each attempting to 
evaluate the impact of the grants on student outcomes after high school such as credits earned, 
GPA, and persistence. There have also been a growing number of studies that use strong quasi- 
experimental designs. This section reviews a selection of eight external studies, four studies 
from the PBS Demonstration, and findings from Opening Doors Louisiana, a program that 
targeted student-parents who were eligible for Temporary Assistance for Needy Families. While 
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these studies do not cover exhaustively all of the research conducted in the literature, they are 
fairly representative of the findings in the field. Appendix Table F. 1 presents a summary of the 
intervention design in each of the studies. While the studies vary in the targeted student 
populations, most evaluate outcomes for traditionally aged students and students who are 
eligible for additional financial aid for at least two semesters. The maximum total additional 
grant amount ranges from a low of $215 to a high of $17,500 per student. Collectively, the 
studies represent over 16,400 randomly assigned students and over 54,800 students matched 
through a quasi-experimental design. Table 6.1 reports means by treatment group, impact 
estimates, and standardized impacts adjusted for inflation (that is, differences generated per 
$1,000 in funds, in 2013 dollars) for these studies. 

Findings on Matriculation 

The first panel of Table 6.1 presents findings on matriculation. The findings from the 
CFC-PBS Program are shown in the first three rows for ease of comparison. Since the program 
group represents the pooled performance-based scholarship groups, the last column of the table 
standardizes the estimates into a difference per $1,000 of scholarship offered to facilitate the 
comparison with other estimates. For example, the CFC-PBS Program resulted in a 4.9 
percentage point increase in matriculation, and the average eligible scholarship amount of the 
five performance-based scholarships, Scholarship Types 2 through 6, is $1,400 (or $1,421 in 
2013 dollars). This translates into an impact on matriculation of 3.4 percentage points per 
$1,000 in 2013 dollars. 

Recall that the CFC-PBS Program is one of the only random assignment evaluations to 
affect matriculation that the authors are aware of. The remaining studies reported in the table 
employ quasi-experimental designs. One study uses changes in the provision of the generous 
Social Security Student Benefit (SSSB) Program for college students to explore the effect of aid 
on attending college before age 23. 1 The study finds that students who were eligible for the 
SSSB Program were 18.2 percentage points more likely to enroll in college than students who 
were not eligible for the additional aid. 2 The last column shows that $1,000 in aid results in a 


'Dynarski (2003). 

2 Dynarski (2003) uses the elimination of the SSSB Program in a difference-in-differences approach (a 
quasi-experimental statistical method) to study the effect of additional aid on college enrollment. Table 6.1 
shows that students who may be eligible for the SSSB Program funds (the program group) had much higher 
rates of college attendance when the program was in effect compared with enrollment rates after the SSSB 
Program was eliminated (as shown by the 20.8 percentage point increase in enrollment in the last line of the 
second panel). Enrollment patterns exhibited a small change (a 2.6 percentage point increase) for those students 
who were not eligible for the program. Thus, the impact of the SSSB Program is 18.2 percentage points (equal 
to 20.8 minus 2.6). 
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The Performance-Based Scholarship Demonstration 
Table 6.1 

Postsecondary Outcomes Across Prior Research on Financial Incentives 

Cash for College Performance-Based Scholarship Study 

Sample Program Control Standard Percentage Difference 

Outcome Size Group Group Difference Error Change per $1,000 


Matriculation (%) 


PBS California 

Enrolled in any college 4,642 

Enrolled in any 2-year college 4,642 

Enrolled in any 4-year college 4,642 

Social Security Student Benefit Program 11 
Enrolled in any college (SSI student benefits) 2,882 

Enrolled in any college (no SSI student benefits) 1,104 

Difference-in-differences 


Cal Grant Program 13 

Enrolled in any college (GPA just above Cal Grant A cutoff) 
Enrolled in any college (GPA just below cutoff) 
Difference-in-differences 


Averase credits earned in first year 


Project STAR C 


Student Support Program (SSP) 

1,418 

Student Fellowship Program (SFP) 

1,418 

Combined program (SFSP) 

1,418 

University of Amsterdam 1 ' 


Low incentive 

166 

High incentive 

165 


89.3 

84.4 

4 Q *** 

1.1 

5.8 

3.4 

47.9 

43.2 

4 7 *** 

1.6 

10.9 

3.3 

42.8 

42.8 

0.0 

1.6 

0.1 

0.0 


56.0 

50.2 

5.8 

- 

11.6 

- 

35.2 

47.6 

-12.4 

— 

-26.1 

- 

20.8 

2.6 

18.2 * 

9.6 

— 

2.0 


87.5 

90.1 

- 2.6 

- 

-2.9 

- 

84.0 

90.3 

-6.3 

- 

-7.0 

- 

3.5 

- 0.2 

3.7 

3.0 

— 

0.8 


2.4 

2.4 

0.1 

0.1 

2.3 

- 

2.4 

2.4 

0.0 

0.1 

-0.5 

0.0 

2.5 

2.4 

0.1 

0.1 

3.9 

0.0 


31.6 

33.2 

-1.5 

3.4 

-4.6 

-5.5 

32.7 

33.2 

-0.4 

3.4 

-1.3 

-0.5 
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Table 6.1 (continued) 


Sample Program Control Standard Percentage Difference 

Outcome Size Group Group Difference Error Change per $1,000 


Average credits earned in first year (continued) 


Opening Doors Louisiana 6 537 

PBS California f 4,642 

PB S New Mexico 1 ,08 1 

PBS New York 1,502 

PBS Ohio 2,285 

West Virginia PROMISE 8 12,91 1 

Wisconsin Scholars ’ 1 1,147 

Average GPA in first year 

Project STAR 6 

Student Support Program (SSP) 1,399 

Student Fellowship Program (SFP) 1,399 

Combined program (SFSP) 1,399 

Opportunity Knocks 1 1,203 

Foundations for Success J 

Service group 2,078 

Service Plus group 2,086 

Opening Doors Louisiana 6 391 

PBS California f 2,324 

PBS New Mexico 1,050 

PBS New York 1,390 

West Virginia PROMISE 8 12,91 1 

Wisconsin Scholars ' 1 1,147 


11.0 

7.7 

3 3 *** 

0.8 

43.7 

1.4 

8.9 

7.9 

Y Q *** 

0.4 

12.9 

0.7 

25.7 

24.8 

0.9 

0.6 

3.7 

0.4 

16.3 

15.5 

0.9 * 

0.5 

5.6 

0.2 

15.6 

13.9 

| 7 *** 

0.4 

12.1 

0.9 

27.8 

26.2 

l ft *** 

0.1 

6.0 

0.4 

26.4 

26.0 

0.5 

0.4 

1.9 

0.1 


1.81 

1.79 

0.01 

0.1 

0.6 

- 

1.75 

1.79 

-0.04 

0.1 

- 2.2 

- 0.01 

1.96 

1.79 

0.17 * 

0.1 

9.4 

0.03 

2.54 

2.52 

0.02 

0.0 

0.8 

0.00 

1.90 

1.88 

0.02 

NA 

1.1 


2.00 

1.88 

0.12 ** 

NA 

6.4 

0.07 

2.11 

1.81 

0.30 *** 

0.1 

16.8 

0.13 

2.31 

2.27 

0.04 

0.0 

1.8 

0.03 

2.78 

2.75 

0.03 

0.1 

1.2 

0.02 

2.57 

2.57 

0.00 

0.1 

0.2 

0.00 

2.92 

2.85 

0 Q7 *** 

0.0 

2.32 

0.02 

2.60 

2.50 

0.00 

0.1 

0.0 

0.00 
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Table 6.1 (continued) 


Sample Program Control Standard Percentage Difference 

Outcome Size Group Group Difference Error Change per $1,000 

Second-year persistence 1 " (%) 

Foundations for Success J 


Service group 

2,078 

62.9 

61.0 

1.9 

NA 

3.1 

- 

Service Plus group 

2,086 

66.3 

61.0 

5 3 *** 

NA 

8.7 

3.3 

Opening Doors Louisiana 6 

537 

30.1 

22.9 

7 2 ** 

3.6 

31.6 

3.0 

PBS California* 

4,642 

81.4 

79.0 

2.4 * 

1.3 

3.0 

1.7 

PBS New Mexico 

1,081 

77.6 

78.5 

-0.9 

2.5 

- 1.2 

-0.4 

PBS New York 

1,502 

61.9 

60.7 

1.2 

2.5 

2.0 

0.4 

PBS Ohio 

2,285 

60.5 

59.2 

1.3 

2.1 

2.2 

0.7 

Wisconsin Scholars ’ 1 

Averase credits earned in second year 1 

1,500 

90.8 

89.1 

1.7 

1.6 

1.9 

0.4 

University of Amsterdam 11 

Low incentive 

166 

23.9 

26.1 

- 2.2 

3.6 

-8.4 

-7.9 

High incentive 

165 

25.0 

26.1 

- 1.1 

3.7 

-4.2 

-1.3 

Opening Doors Louisiana 6 

537 

13.7 

10.0 

3 7 *** 

1.2 

36.9 

1.5 

PBS New Mexico 

1,081 

46.5 

44.3 

2.2 * 

1.3 

5.1 

1.1 

PBS New York 

1,502 

26.7 

25.6 

1.1 

0.9 

4.3 

0.3 

PBS Ohio 

2,285 

24.8 

22.4 

o 4 *** 

0.8 

10.6 

1.2 

Earned any degree or certificate by third year (%) 

PBS Ohio 

2,285 

26.9 

23.3 

3.5 * 

1.8 

15.1 

1.8 

Wisconsin Scholars ’ 1 

1,500 

3.2 

2.9 

0.3 

0.9 

10.3 

0.1 
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Table 6.1 (continued) 


SOURCES: MDRC calculations using enrollment data from Social Security Student Benefit Program (Dynarski, 2003); enrollment data from 
Cal Grant Program (Kane, 2003); credits earned and GPA data from Project STAR (Angrist, Lang, and Oreopoulos, 2009); credits earned data 
from University of Amsterdam (Leuven, Oosterbeek, and van der Klaauw, 2010); credits earned and GPA from West Virginia PROMISE (Scott- 
Clayton, 2011); credits earned, GPA, enrollment, and degree attainment data from Wisconsin Scholars (Goldrick-Rab, Harris, Benson, and 
Kelchen, 2011); GPA data from Opportunity Knocks (Angrist, Oreopoulos, and Williams, 2010); GPA and enrollment data from Foundations for 
Success (MacDonald et al., 2009); National Student Clearinghouse data; and transcript data from Delgado Community College, Louisiana 
Technical College, the California Community College Chancellor’s Office, the University of New Mexico, Borough of Manhattan Community 
College, Hostos Community College, and the Ohio Board of Regents. 

NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

The last column shows the impact per $1,000 in 2013 dollars. NA = not applicable. 

MDRC estimates are adjusted by research cohort and campus. 

a SSI is Supplemental Security Income. The “SSI student benefits” row represents the period in which additional financial support was 
available through the Social Security Student Benefit Program (SSSBP). SSSBP provided up to $6,700 (in 2000 dollars) of aid to students whose 
parents were Social Security beneficiaries and were deceased, disabled, or retired. The “no SSI student benefits” row represents the period after 
the program was eliminated. The “program group” column represents the estimates for students with deceased parents (those eligible for 
SSSBP), while “control group” column estimates are for students whose parents were not deceased. Estimates represent attendance in college 
before age 23 and are based on the difference-in-differences analysis presented in Table 1 and Table 2 in Dynarski (2003). 

b The “GPA just above Cal Grant A cutoff’ row represents the group of students with high school GPAs that just exceed the thresholds for the 
Cal Grant A program in 1999. The “GPA just below cutoff’ represents the group of students with high school GPAs that are just below the 
thresholds for the Cal Grant A program in 1999. The program group represents the group of students who were financially eligible for Cal Grant 
A, while the control group represents students who did not meet the income and asset limits required by the Cal Grant A program in 1999. The 
difference between the program group estimates and the control group estimates therefore represents the effect of Cal Grant A financial 
eligibility. Estimates are taken from the 1999 panel in Table 5 in Kane (2003). 

C SSP is a student service program only, without a financial incentive component. SFP is a financial incentive without student services. The 
SFSP program combined student services with a financial incentive. The control group and program group means may be estimates, and not the 
true means, based on the impact estimate and other information provided in the research paper. Differences were detected by gender, with 
impacts being significant at the 5 percent level for women in the combined group for both average credits earned in the first year and average 
GPA in the first year. Sample size represents the analysis sample used to analyze treatment effects on second-year outcomes. Estimates are taken 
from Table 6 in Angrist, Lang, and Oreopoulos (2009). 

^Estimates are taken from Table 4 and Table 8 in Leuven, Oosterbeek, and van der Klaauw (2010). 

e Only the first two cohorts are shown for Opening Doors Louisiana. 
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Table 6.1 (continued) 


NOTES (continued): f Credits earned and GPA are based on intent-to-treat averages using data for California community colleges. The PBS 
California program group consists of performance-based scholarships only (Scholarship Types 2 through 6). 

^Estimates are based on ordinary least squares difference-in-differences estimates from Table 3 in Scott-Clayton (2011). 
h The Wisconsin Scholars report uses analytic subsample 1 estimates for credits earned and full sample estimates for second-year persistence 
and associate degree attainment. All entries are taken from Table 4 in Goldrick-Rab, Harris, Benson, and Kelchen (2011). 

'The control group and program group means may be estimates, and not the true means, based on the impact estimate and other information 
provided in the research paper. Estimates are taken from Table 4b in Angrist, Oreopoulos, and Williams (2010). 

Tor the Foundations for Success report, only students in the Service Plus group were eligible for the incentive grant. GPA shown is for the 
second semester of the first year only. Estimates are taken from Table 6-2 and Table 7-4 in MacDonald et al. (2009). 

k Figures for Foundations for Success represent retention rates from semester 2 to semester 3, while the remaining entries reflect registration 
the third semester, or fall-to-fall persistence. 

'Figures for the University of Amsterdam represent credits earned in the second year, while the remaining entries reflect cumulative credits 
over the first two years. 



2.0 percentage point increase in enrollment. 3 Further, another study using a quasi-experimental 
design does not find that eligibility for Cal Grant A results in a statistically significant impact on 
enrollment. 4 

Overall, the estimates from the CFC-PBS Program appear to be largest for inducing 
enrollment at community colleges when the impact per $1,000 in additional aid is examined. 
This finding indicates that relatively small scholarship amounts could be effective in increasing 
matriculation at community colleges if the scholarships were well designed, targeted effectively, 
and disbursed early enough to affect students’ decisions about going to college. As noted 
earlier, the June disbursement for the CFC-PBS Program was likely too late to affect decisions 
to matriculate at four-year institutions, as such institutions often require acceptances by April. 

Findings on Credits Earned 

The second panel of Table 6.1 examines credits earned in the first year of college. The 
first set of rows reports findings from Project STAR and the University of Amsterdam. Project 
STAR offered student services (specifically, peer advising), an incentive, or a combination of 
both to entering college freshmen of a large Canadian university. Students who did not perform 
well in high school were eligible for the largest incentive amount if they managed to become 
top students in college. 5 The University of Amsterdam program sought to encourage college 
students in economics and business to pass their first-year classes by offering them an incentive. 
The incentive, paid at the end of the year, was provided in a high amount (around $834 in 2013 
dollars) and a low amount (around $278 in 2013 dollars) and was randomly assigned to two 


3 Dynarski’s (2003) preferred specification, which controls for covariates and adjusts for classification 
error, results in an impact estimate of a 2.7 percentage point increase in enrollment per $1,000 (in 2013 dollars) 
in grant aid. 

4 Kane (2003) uses a difference-in-differences approach to study the effect of Cal Grant A eligibility on 
college enrollment. Cal Grant A can be used at four-year institutions in California. Students who met the 
income and asset limits for Cal Grant A in 1999 (the program group) — enrolled at higher rates when they had 
high school GP As just above the cutoff compared with similar students who had high school GP As just below 
the cutoff (as shown by the 3.5 percentage point difference in the last line of the third panel of Table 6.1). High 
school GPAs matter less for enrollment patterns when students’ income or assets exceeded the Cal Grant A 
limits (as shown by the -0.2 percentage point difference in enrollment for these students in the last line of the 
third panel of Table 6.1). The impact of being eligible for the Cal Grant A program is a statistically 
insignificant increase in enrollment of 3.7 percentage points. 

incentives were paid based on high school quartiles, so that students received the highest pay relative to 
their improvement in GPA from their high school performance. That is, students who graduated high school in 
the lowest quartile could receive $5,000 for a B average or $1,000 for a C+ average, while students in the third 
quartile were offered $5,000 for A-s and $1,000 for Bs. 
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separate program groups. The entries in Table 6. 1 show that neither program had much effect 
on credit accumulation in the first year. 6 

The next set of entries shows findings from the Opening Doors Louisiana study and a 
number of studies that are part of the PBS Demonstration. The entries show that all of the 
programs (with the exception of New Mexico’s, which is the only four-year institution in the 
PBS Demonstration) resulted in increases in credits earned in the first year (ranging from 0.9 to 
3.3 credits, for increases of 6 percent to 44 percent). Opening Doors Louisiana yielded the 
largest impact, of 1.4 credits per $1,000. 

The West Virginia PROMISE program is a merit-based state aid program that offers 
free tuition and fees at any state public institution for eligible students who maintain a minimum 
GPA of 3.0 and a full-time course load. 7 This program resulted in an increase of 1.6 credits 
earned after one year (an increase of 6 percent, or about one-half credit per $1,000). The 
Wisconsin Scholars program randomly offered an extremely generous need-based grant (worth 
$3,500 per year for up to five years) to freshmen Pell Grant recipients attending public 
universities in Wisconsin. 8 This program did not materially change credit accumulation in the 
first year. 

GPA in First Year and Second-Year Outcomes 

The next panel of the table reports on GPA in the first year. Overall, large changes in 
GPA have not been observed in various studies. This may not be surprising, as GPA represents 
the cumulative experience of students and their decisions, which may be difficult to alter with 
modest interventions. Opening Doors Louisiana induced the largest change in GPA — 0.30 
points, or an increase of 0.13 points per $1,000. The next largest impact was generated by the 
combined incentive program under Project STAR, which induced a change of 0.17 points, or 
0.03 points per $1,000. 9 The merit program in West Virginia improved GPA by 0.07 points. 10 

The next two panels examine second-year persistence (or persistence to the third term) 
and average credits earned in the second year. Only Foundations for Success’s combined group, 


6 Separate analyses (not shown in the table) suggest that effects exist for women assigned to the combined 
group of services in Project STAR. See Angrist, Lang, and Oreopoulos (2009). 

7 Students were allowed a slightly lower GPA benchmark of 2.75 in their first year. The average award was 
$3,755 (in 2013 dollars) for the first year, and students received an average of about $12,950 (in 2013 dollars) 
over four years. See Scott-Clayton (2011). 

8 Goldrick-Rab, Harris, Benson, and Kelchen (2011). While program group members needed to meet 
satisfactory academic progress measures that included a 2.0 GPA (the requirements for continued eligibility for 
the federal Pell Grant), the program did not include additional activities or requirements to make the academic 
benchmarks (satisfactory academic progress) salient. 

9 Angrist, Lang, and Oreopoulos (2009). 

10 Scott-Clayton (2011). 
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Opening Doors Louisiana, and the CFC-PBS Program affected second-year persistence, with 
the Foundations for Success program increasing persistence by the largest amount — 3.3 
percentage points per $1,000." Interestingly, this represents effects observed after the program 
ended for Opening Doors Louisiana (a one-year program) and possibly for the CFC-PBS 
Program as well. Chapter 4 in this report suggests that the CFC-PBS impact likely reflects 
Scholarship Type 3 ($500 per term over two terms) and Scholarship Type 4 ($2,000 in three 
payments over two semesters). For credits earned in the second year, Opening Doors Louisiana 
continues to generate the largest impact (1.5 credits per $1,000), followed by PBS Ohio and 
PBS New Mexico. 

The final panel in Table 6.1 examines degree or certificate attainment by the third year. 
Only the studies of the PBS Ohio program and the Wisconsin Scholars program report on this 
metric. The PBS Ohio program increased degree or certificate attainment by 3.5 percentage 
points, or 1.8 percentage points per $1,000. 12 

Where Does the CFC-PBS Program Stand? 

Overall, the analysis suggests that the findings from the CFC-PBS Program compare 
favorably with those from other studies in the PBS Demonstration and other studies in the 
literature. This finding is notable, as the CFC-PBS Program may have had the lightest 
interaction with students because the scholarship is completely portable and the program was 
implemented by a statewide partnership, while other programs in the PBS Demonstration are 
based at institutions. When compared with other statewide programs like West Virginia 
PROMISE and Wisconsin Scholars, the CFC-PBS program fares in the middle. Findings from 
the Opening Doors Louisiana study — the original study evaluating perfonnance-based 
scholarships — seem to show the largest impacts to date. 

In general, impacts for perfonnance-based scholarship programs are more likely to be 
positive and statistically significant than those from other research on scholarships found in the 
literature. This may reflect the targeting of programs; on average, students in the PBS 
Demonstration have one or more risk factors for not completing college, such as being low- 
income, older, parents, and so on, which may contribute to the larger effect of the contingent 


"MacDonald et al. (2009). The Foundations for Success project targeted incoming freshmen at three 
Canadian colleges. The program randomly assigned participants to two program groups: the Service Group 
(which provided case management where students were required to complete 12 hours of approved activities 
that could include tutoring and assessments such as the Myers-Briggs Type Indicator) or the Service Plus group 
(in which students were required to complete the same case management objectives but were also eligible for a 
fellowship after completing the 12-hour requirement). The fellowship, worth $750 per semester (or $2,250 
total), was paid at the start of the following semester for three semesters. 

12 The Wisconsin Scholars entry represents associate’s degree attainment for the first cohort. 
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grant on academic outcomes. In addition, the treatment contrast — what the intervention 
provides above and beyond the status quo — could be greater for these programs. 


Implications for Innovations in Financial Aid Policy 

The findings presented in Chapter 5 suggest that performance-based scholarships may be less 
costly than traditional grant programs because fewer dollars are disbursed than are offered as a 
result of the conditional nature of the scholarship. Given that the benchmarks for performance- 
based scholarships are similar to those required for most aid programs, some may wonder 
whether the strategy of tying such federal and state payments more closely to achievement 
would lead to lower costs. While it is unknown whether similar impacts could be generated by 
such changes, these changes have potential implications that do not arise in the performance- 
based scholarships studied in this report. 

Implications for National Policy 

There are a few considerations to keep in mind when considering alterations to the Pell 
Grant. First, the scholarships studied in the PBS Demonstration are all paid in addition to 
existing financial aid, including aid from the Pell Grant, ensuring that most students receive a 
net increase in funds. 13 In fact, the vast majority of students in the PBS Demonstration, 
including those in the CFC-PBS Program, are eligible for Pell Grants, which provides the 
foundation of most financial aid packages in the studies. As discussed in Chapter 4, changing 
the base foundation of aid can have unexpected effects, because the behavioral response to aid 
is likely nonlinear. That is, the behavior change in response to a change in Pell Grant 
disbursement criteria may be different from that observed in the PBS Demonstration because 
Pell is generally a much larger grant and one of the first components of financial aid in a low- 
income student’s aid package. Second, altering the availability of the Pell Grant could have a 
chilling effect on enrollment, particularly for matriculation among at-risk groups. Perceiving 
that there is a substantial amount of aid to help pay for college at the beginning of the year likely 
helps relieve the stress of worrying about finances for low-income students. Removing this 
perception could result in fewer low-income students gaining access to the higher education 
system. 


While the current report and past reports have not found evidence of unintended 
consequences, adding performance criteria and changing the disbursement policy to the Pell 
Grant program could result in undesirable behavior, as described above. The potential for these 

1 3 A small number of students in several PBS Demonstration studies had their financial aid packages 
altered in order to receive the performance-based scholarship. Most of these changes resulted in decreases in 
the loan amounts awarded. See Cha and Patel (2010); Miller, Binder, Harris, and Krause (2011); Patel and 
Valenzuela (2013). 


116 



unintended consequences would be exacerbated by the scale of the program. In addition to 
increasing the uncertainty of funding for students, changing this system could also create much 
upheaval for colleges themselves, as they are unlikely to have other funds to assist students who 
may need a stable source of aid. 

Implications for State Policy in California 

Budgetary shortfalls and the recent recession spanning 2007 to 2009 hurt California’s 
community colleges and universities. The demand for postsecondary education increased at the 
same time that the state’s support for the three college systems (community colleges, California 
State Universities, and the University of California) was reduced. Although the passage of 
California Proposition 30 during the 2012 election cycle helped to stabilize the financial 
situation in the state, public education funding has only returned to levels comparable with those 
before the economic collapse. 14 

In such a budgetary situation, accompanied by increased demand for financial aid, 
systematically studying the effect of randomly implementing performance-based aid or 
exploring alternative distribution patterns may be prudent policy. For example, policymakers 
may be able to experiment with perfomiance criteria by lowering the high school GPA 
threshold and permitting students who would not otherwise be eligible to earn performance- 
based Cal Grants. 15 This could potentially increase the proportion of low-income students who 
matriculate. Alternatively, instead of raising the Cal Grant high school GPA threshold (as was 
proposed during the 2012-2013 budget discussions), the original GPA level could be 
maintained if the scholarship were made performance-based. 16 In this way, more students would 
be served despite restrictions to eligibility. As mentioned earlier in the discussion of scholarship 
take-up rates, program group students received a proportion of the total scholarship amount 
available to them. All else being equal, scholarships with more perfonnance requirements cost 
less than scholarships with fewer perfonnance requirements. While the addition of perfonnance 
requirements means that students are likely to receive less money on average, it also means that 
the scholarship can be offered to more students in need. 

Implications for Scholarship Providers 

Private scholarship providers often give scholarships based on various eligibility criteria 
that may or may not include financial need. Indeed, some of these scholarships are administered 


14 Calefati and Richman (2013). 

15 While there are some performance criteria in the existing Cal Grant program, they vary, and their 
application can be delayed based on institutional requirements. Performance -based programs in this context 
imply a greater frequency and earlier checks on performance adherence. 

16 Taylor (2012). 
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somewhat haphazardly . 17 They often do not have specific goals or, as with merit-based 
scholarships, they go to students who already have a high chance of academic success. 

Scholarship providers that are able to experiment with performance-based aid programs 
could help answer questions about how perfonnance-based scholarships affect students and 
how these scholarships might help the providers to accomplish their goals. Perfonnance-based 
scholarships could help traditional scholarship programs maximize the amount of money they 
are able to offer, because students are offered the opportunity to earn more scholarship dollars 
based on the demonstration of current perfonnance, rather than relying on past perfonnance as a 
predictor. But here, too, there could be trade-offs if dependable, non-perfonnance-based money 
often makes it possible for students to enroll who otherwise would not. Further experimentation 
is necessary to understand the true effects. 

Next Steps for Project 

This report presents early findings of the CFC-PBS Program. A future report will 
discuss longer-tenn academic outcomes for all scholarship types and provide the final cross-site 
results on the perfonnance-based scholarship programs over the duration of the project. 


17 Woo and Choy (2011). 


118 



Appendix A 

Technical Appendix 




Power 


Minimum detectable effects were calculated for the study using one-year retention rates at both 
four-year universities (79 percent) and community colleges (68 percent). The proposed first- 
cohort sample (700 program group members and 1,400 control group members) was estimated 
to be able to detect a 5 percentage point difference at community colleges and a 4 percentage 
point difference at four-year universities. The proposed total sample of 4,620 (1,540 program 
group members and 3,080 control group members) was estimated to able to detect a 3.6 
percentage point difference at community colleges and a 3.1 percentage point difference at four- 
year universities. 


Analysis Sample 

Figure A.l presents a visual representation of how students were randomly assigned to the 
California Cash for College Performance-Based Scholarship (CFC-PBS) Program. It expands 
upon the discussion of the study intake in Chapter 2. 1 

A total of 15,420 students who attended Cash for College workshops in one of the Los 
Angeles, Far North, Kern County, or Capital regions expressed their interest in being a part of 
the CFC-PBS Program and were assessed for eligibility in the study. Nearly 6,000 of these 
students were excluded from the study because they did not meet the eligibility criteria. 

Around 49 percent (or 2,820) of excluded students were not found in the California 
Student Aid Commission’s financial aid database, either because they did not complete a 
FAFSA or AB 540 form or because they had some information on the informed consent form 
or Exit Survey that precluded them from being matched to the financial aid database. 

An additional 4 1 percent of students were deemed ineligible because they were above 
the Cal Grant (state financial aid) income thresholds. 2 Over 550 students (around 10 percent of 
those excluded) were ineligible because they did not submit an informed consent form, did not 
complete the mandatory Exit Survey at their Cash for College workshop, or were not a high 
school senior under the age of 20 at the time of their workshop. 

Around 40 students were excluded because they did not attend a sufficiently large Cash 
for College workshop. In 2009, workshops with fewer than 8 attendees were ineligible, and in 
2010, workshops with fewer than 12 attendees were ineligible. MDRC worked with CSAC to 


'The flow diagram adheres to the requirements of the Consolidated Standards of Reporting Trials 
(CONSORT) on displaying how the analysis sample was derived. See Schulz, Altman, and Moher (2010). 

2 See Chapter 2 for an example income ceiling for Cal Grant receipt for a family of four. 
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The Performance-Based Scholarship Demonstration 
Appendix Figure A.l 
CONSORT Statement Flow Diagram 


Cash for College Performance-Based Scholarship Study 



Analyzed 
(n = 279) 


Analyzed 
(n = 264) 


Analyzed 
(n = 273) 


Analyzed 
(n = 276) 


Analyzed 
(n = 276) 


Analyzed 
(n = 272) 


Analyzed 
(n = 3,281) 


(continued) 




















Appendix Figure A.l (continued) 


NOTES: The flow diagram adheres to the requirements of the Consolidated Standards of Reporting Trials (CONSORT) on displaying how the 
analysis sample was derived (see Schulz, Altman, and Moher, 2010). No students in the study were lost to follow-up, so the analysis sample is 
identical to the allocated sample. 

A total of 289 Cash for College workshops participated in the study. 

Each scholarship group was randomized to have a corresponding control group, resulting in six control groups, where the size of the 
control group was two times that of the program group. 

There were 239 undocumented, AB 540 students who participated in the study but were excluded from the analysis sample. 



combine the smaller workshops of similar regions in an attempt to limit the number of students 
lost to this eligibility criterion. 

By the end of the eligibility determination process, all students had signed an informed 
consent form, had completed the Exit Survey, had filled out a FAFSA or AB 540 worksheet, 
were high school seniors and under age 20, were below the Cal Grant income thresholds, and 
had attended a sufficiently large Cash for College workshop. 

Finally, 239 undocumented, AB 540 students were excluded from the analysis sample 
because data collection for these students was incomplete as a result of the absence of Social 
Security numbers and poor matches by name and date of birth. 


Random Assignment 

The first step of the random assignment process was to separate the eligible 9,665 students by 
region and workshop and determine the number of scholarships to be disbursed in each of the 
regions (two regions in 2009 and four regions in 2010). The probability of scholarship receipt in 
a given region was then calculated based on the sample size of that region. Using this probabil- 
ity and the size of an individual workshop, a corresponding number of students were randomly 
assigned to the program group. Twice as many students were randomly assigned to the control 
group, while all remaining students were assigned to the nonstudy group. Program group and 
control group students were then randomized to one of six scholarship types, such that there 
were six program groups of around 285 students for each scholarship type and a pooled control 
group of 3,440 students. None of the 5,160 students assigned to either the program group or 
control group withdrew their participation in the study, and thus, no students were lost to 
follow-up. A total of 239 undocumented, AB 540 students who participated in the study were 
excluded from all analyses, leaving a total analysis sample of 4, 921 students. 


Matching to the National Student Clearinghouse 

An analysis of baseline characteristics by found status suggests that students who are not found 
in the National Student Clearinghouse (Clearinghouse) are more likely to be Hispanic/Latino, to 
be the first person in their family to attend college, to have parents whose highest level of 
educational attainment is high school or a General Educational Development (GED) certificate, 
and to speak a language other than English regularly in their home — all characteristics that 
may be associated with lower college-going rates (see Appendix Table A.1). 3 In addition, 
program group members were more likely to be found than control group students, indicating 


3 Engle (2007); Choy (2001); Terenzini et al. (1996); Klein, Bugarin, Beltranena, and McArthur (2004). 
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The Performance-Based Scholarship Demonstration 
Appendix Table A.l 

Selected Characteristics of Sample Members at Baseline, 
Based on Whether Found by the Clearinghouse 


Cash for College Performance-Based Scholarship Study 



Analysis 


Not 

Standard 

Characteristic 

Sample 

Found 

Found 

Difference 

Error 

Gender (%) 

Male 

39.8 

39.6 

43.0 

-3.4 

2.9 

Female 

60.2 

60.4 

57.0 

3.4 

2.9 

Race/ethnicity a (%) 

Latino 

60.9 

60.5 

66.3 

-5.8 ** 

2.5 

White 

20.3 

20.2 

21.4 

- 1.2 

1.8 

Black 

3.8 

4.0 

1.1 

2.8 ** 

1.1 

Asian or Pacific Islander 

11.0 

11.2 

7.4 

39 ** 

1.8 

Other 

4.0 

4.0 

3.7 

0.3 

1.1 

High school senior (%) 

100.0 

100.0 

100.0 

0.0 

0.0 

First person in family to attend college (%) 

53.7 

53.1 

62.6 

_g 5 *** 

2.9 

Highest degree/diploma earned by either parent (%) 

Not a high school graduate 34.9 

34.7 

37.3 

- 2.6 

2.7 

High school diploma or GED certificate 

31.1 

30.7 

37.4 

-6.7 ** 

2.7 

Some college or associate’s degree 

22.7 

22.9 

20.1 

2.8 

2.4 

Bachelor’s degree or higher 

11.3 

11.7 

5.2 

6.6 *** 

1.8 

Language other than English spoken regularly in 

home (%) 

60.7 

60.6 

61.7 

- 1.1 

2.4 

High school cumulative GPA (average) 

2.9 

2.9 

2.5 

Q ^ *** 

0.0 

Motivation to apply for financial aid b (average) 

Relative Autonomy Index 

1.4 

1.4 

1.1 

0.3 

0.2 

External regulation subscale 

6.7 

6.8 

6.7 

0.1 ** 

0.0 

Introjected regulation subscale 

4.2 

4.1 

4.5 

-0.3 ** 

0.1 

Identified regulation subscale 

6.3 

6.3 

6.2 

0.0 

0.1 

Integrated regulation subscale 

6.4 

6.4 

6.3 

0.1 

0.1 

Sample size 

4,921 

4,601 

320 




(continued) 
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Appendix Table A.l (continued) 


SOURCES: MDRC calculations using Exit Survey (Baseline Information Form) data and National 
Student Clearinghouse data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Missing values are not included in individual variable distributions. 

Distributions may not sum to 100 percent because of rounding. 

a Students who identify as Latino are shown only in the Latino category, even if they selected more 
than one race. Students who selected more than one race and are not Latino are considered multiracial. 
The “other” category comprises American Indian/Alaska Native or multiracial students, or students of 
some other race/ethnicity. 

b Motivation to apply for financial aid is defined using the Relative Autonomy Index (RAI), which has 
a range of -18 to 18, where a higher value represents greater autonomous motivation. The RAI is 
calculated as a weighted average: RAI = [Externalx(-2)] + [Introjectedx(-l)] + [Identifiedx(l)] + 
[lntegratedx(2)]. See Deci, Koestner, and Ryan (2001). 


higher enrollment among program group students at institutions reporting to the Clearinghouse. 
To rule out potential bias from the possibility that control group students enrolled in different 
types of institutions (types that do not report to the Clearinghouse), infomiation on enrollment 
from the survey was analyzed. The analysis suggests that control group students were equally 
likely to register at institutions that do not report to the Clearinghouse as program group 
students, which suggests that control group students were less likely to be found by the Clear- 
inghouse because they were truly not attached to an educational institution. 


Estimation Equation 

The primary analytic method used to determine program impacts is comparing average out- 
comes for the pooled program group and comparison group members, using standard statistical 
tests such as the t-test. More fonnally, estimates in the report are obtained using ordinary least 
squares (OLS) regressions of the fonn 

E t = a + Y.s fis T is + P T iCFC + VQ + Zj (PjRj + £; 

Where: 


Ej = represents the educational attainment of individual i, 

T; s = indicator equal to 1 if the individual i was randomly assigned to the per- 
fonnance-based scholarship type s and equal to 0 otherwise, 

T;cfc = indicator equal to 1 if the individual i was randomly assigned to the 
Cash for College scholarship type and equal to 0 otherwise, 
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C; = indicator equal to 1 if the individual i was part of cohort II, 


Ri = indicator representing the region j from which the student was randomly 
selected (while individuals were randomized at the workshop level, specifica- 
tions at the region level are used to represent the randomization pool), 

8; = a random error tenn, and 

a, (3, 5, y, and cp = coefficients to be estimated. 

The coefficients of interest are [3, which represents the effect of assignment to the CFC-PBS 
program group on the outcome of interest, and 5 S , which represents the effect of assignment to 
one of the five performance-based scholarships groups (where one group is omitted). Because 
the students are randomly assigned to receive the treatment (or not), the background characteris- 
tics (including unobserved characteristics such as motivation and determination) of the two 
groups will be the same, on average. As a result, OLS estimation of (3 and 5 S will provide an 
unbiased estimate of the “intent-to-treaf ’ effect, and it is not necessary to control for other 
student characteristics. 


Survey Response Analysis 

A follow-up survey was administered to all CFC-PBS sample members in the spring of the year 
following random assignment. The overall response rate for the analysis sample was 80 percent, 
which was equal to MDRC’s prespecified target response rate. The response rate for the 
analysis sample of program group students was 85 percent, compared with 77 percent for the 
analysis sample of control group students. 

Program Group Versus Control Group Representation Among Survey 
Respondents 

An omnibus F-test was performed to examine whether statistically significant differ- 
ences in baseline characteristics existed between the analysis sample program group and control 
group members who responded to the follow-up survey (see Appendix Table A.2). The p-value 
of this test was 0.69, indicating that background characteristics do not predict treatment status 
among survey respondents. Therefore, this report does not weigh survey responses, as the 
impacts are unlikely to be biased. 

Nonresponse Bias 

An omnibus F-test was performed to examine whether statistically significant differ- 
ences in baseline characteristics existed between those analysis sample members who respond- 
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The Performance-Based Scholarship Demonstration 
Appendix Table A.2 

Selected Characteristics of PBS Survey Respondents at Baseline, 

by Research Group 


Cash for College Performance-Based Scholarship Study 


Characteristic 

Analysis Sample 
Respondents 

Program 

Group 

Control 

Group 

Standard 

Error 

Gender (%) 

Male 

38.4 

38.0 

38.6 

1.6 

Female 

61.6 

62.0 

61.4 

1.6 

Race/ethnicity a (%) 

Latino 

60.9 

60.3 

61.2 

1.4 

White 

20.2 

20.1 

20.3 

1.0 

Black 

3.8 

3.7 

3.8 

0.6 

Asian or Pacific Islander 

11.0 

10.9 

11.1 

1.0 

Other 

4.1 

5.0 

3.6 ** 

0.7 

High school senior (%) 

100.0 

100.0 

100.0 

0.0 

First person in family to attend college (%) 

53.6 

54.1 

53.3 

1.6 

Highest degree/diploma earned by parent (%) 

Not a high school graduate 

35.6 

35.0 

35.9 

1.5 

High school diploma or GED 

30.5 

29.8 

30.9 

1.5 

Some college or associate’s degree 

22.3 

23.7 

21.5 * 

1.4 

Bachelor’s degree or higher 

11.7 

11.5 

11.7 

1.1 

Language other than English spoken regularly in 

home (%) 

61.8 

60.6 

62.4 

1.4 

High school cumulative GPA (average) 

2.9 

2.9 

2.9 

0.0 

Motivation to apply for financial aid 1 ’ (average) 

Relative Autonomy Index 

1.4 

1.5 

1.4 

0.1 

External regulation subscale 

6.7 

6.7 

6.8 * 

0.0 

Introjected regulation subscale 

4.1 

4.2 

4.1 

0.1 

Identified regulation subscale 

6.3 

6.3 

6.3 

0.0 

Integrated regulation subscale 

6.4 

6.4 

6.4 

0.0 

Sample size 

3,920 

1,394 

2,526 



(continued) 
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Appendix Table A.2 (continued) 


SOURCES: MDRC calculations using Exit Survey (Baseline Information Form) data, Cal Grant data 
provided by the California Student Aid Commission, and responses from the Performance-Based 
Scholarship 12-Month Survey. 

NOTES: To analyze whether baseline characteristics and research group status predicted survey 
response, a likelihood ratio test was performed, which yielded a p-value of 0.69. This suggests that the 
differences in baseline characteristics between program group and control group survey respondents are 
likely to have occurred by chance. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Missing values are not included in individual variable distributions. 

Distributions may not sum to 100 percent because of rounding. 

Calculations for this table used all available data for the 3,920 analysis sample survey respondents. 

a Students who identify as Latino are shown only in the Latino category, even if they selected more 
than one race. Students who selected more than one race and are not Latino are considered multiracial. 
The “other” category comprises American Indian/ Alaska Native or multiracial students, or students of 
some other race/ethnicity. 

b Motivation to apply for financial aid is defined using the Relative Autonomy Index (RAI), which 
has a range of -18 to 18, where a higher value represents greater autonomous motivation. The RAI is 
calculated as a weighted average: RAI = [Externalx(-2)] + [Introjectedx(-l)] + [Identifiedx(l)] + 
[lntegratedx(2)]. See Deci, Koestner, and Ryan (2001). 


ed to the follow-up survey and those who did not (see Appendix Table A.3). The p-value of this 
test was less than 0.01, which is significant at the 1 percent level. This indicates that analysis 
sample respondents differ from nonrespondents; specifically, respondents are more likely to be 
women, have parents who did not graduate from high school, and speak a language other than 
English regularly at home. Nonetheless, it is unlikely that this imbalance in response biases the 
findings because results for respondents using only administrative records are qualitatively 
similar to those of the full sample. This suggests that the patterns of academic outcomes are 
similar for respondents as for the full sample. 
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The Performance-Based Scholarship Demonstration 
Appendix Table A.3 

Selected Characteristics of Sample Members at Baseline, by PBS Survey 
Respondents and Nonrespondents 


Cash for College Performance-Based Scholarship Study 



Analysis 


Non- Standard 

Characteristic 

Sample 

Respondents 

respondents 

Error 

Gender (%) 





Male 

39.8 

38.4 

45 4 *** 

1.7 

Female 

60.2 

61.6 

54.6 *** 

1.7 

Race/ethnicity a (%) 





Latino 

60.9 

61.0 

60.4 

1.5 

White 

20.3 

20.2 

21.0 

1.1 

Black 

3.8 

3.8 

4.0 

0.7 

Asian or Pacific Islander 

11.0 

11.0 

10.9 

1.1 

Other 

4.0 

4.1 

3.7 

0.7 

High school senior (%) 

100.0 

100.0 

100.0 

0.0 

First person in family to attend college (%) 

53.7 

53.6 

54.0 

1.7 

Highest degree/diploma earned by parent (%) 





Not a high school graduate 

34.9 

35.6 

31.8 ** 

1.6 

High school diploma or GED 

31.1 

30.5 

33.6 * 

1.7 

Some college or associate’s degree 

22.7 

22.3 

24.5 

1.5 

Bachelor’s degree or higher 

11.3 

11.6 

10.1 

1.1 

Language other than English spoken regularly in 
home (%) 

60.7 

61.8 

56.0 *** 

1.5 

High school cumulative GPA (average) 

2.9 

2.9 

2 g *** 

0.0 

Motivation to apply for financial aid b (average) 





Relative Autonomy Index 

1.4 

1.4 

1.3 

0.1 

External regulation subscale 

6.7 

6.7 

6.7 

0.0 

Introjected regulation subscale 

4.2 

4.1 

4.3 ** 

0.1 

Identified regulation subscale 

6.3 

6.3 

6.3 

0.0 

Integrated regulation subscale 

6.4 

6.4 

6.4 

0.0 

Sample size 

4,921 

3,920 

1,001 



(continued) 
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Appendix Table A.3 (continued) 


SOURCES: MDRC calculations using Exit Survey (Baseline Information Form) data, Cal Grant data 
provided by the California Student Aid Commission, and responses from the Performance-Based 
Scholarship 12-Month Survey. 

NOTES: To analyze whether baseline characteristics and research group status predicted survey 
response, a likelihood ratio test was performed, which yielded a p-value of less than 0.01. This suggests 
that the differences in baseline characteristics between respondents and nonrespondents are unlikely to 
have occurred by chance. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Missing values are not included in individual variable distributions. 

Distributions may not sum to 100 percent because of rounding. 

“Students who identify as Latino are shown only in the Latino category, even if they selected more 
than one race. Students who selected more than one race and are not Latino are considered multiracial. 
The “other” category comprises American Indian/ Alaska Native or multiracial students, or students of 
some other race/ethnicity. 

b Motivation to apply for financial aid is defined using the Relative Autonomy Index (RAI), which 
has a range of -18 to 18, where a higher value represents greater autonomous motivation. The RAI is 
calculated as a weighted average: RAI = [Extemalx(-2)] + [Introjectedx(-l)] + [Identifiedx(l)] + 
[lntegratedx(2)]. See Deci, Koestner, and Ryan (2001). 
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The Performance-Based Scholarship Demonstration 
Appendix Figure B.l 

Sample CFC-PBS Scholarship Award Letter 
Cash for College Performance-Based Scholarship Study 

<Date> 

<Student First Name> <Student Last Name> 

<Street Address> 

<City>, <State> <Zip Code> 

Workshop ID#: < Workshop ID#> 

Congratulations <Student First Name>, you’ve been awarded a $1,000.00 Cash for College Performance- 
based Scholarship for one semester and are now a Cash for College Scholar! 

You have been selected to join a special group of Cash for College Scholars as part of the Cash for 
College Performance-based Scholarship (CFC-PBS) study. We are pleased to be investing $1,000.00 in 
you and your future, and we are confident you will succeed! In addition to the scholarship award, you 
will have access to a select online community of other scholars. 

This Cash for College Performance-based Scholarship acknowledges that you took important steps to 
pursue higher education that will help you reach your personal and professional goals. 

It’s very easy to claim your scholarship. Beginning June 1 st , visit 

www.calgrants.org/cashforcollegescholarship . The website will provide you with easy to follow instruc- 
tions, including a short video, to claim your award. Because these awards are scarce and in such high 
demand, you must claim your award no later than Tuesday, August 3, 2010. You may lose your 
scholarship in whole or in part if it is not claimed by August 3 rd . The attached sheet contains specific 
instructions about claiming your scholarship, the details of your specific scholarship, the steps you 
must take to receive your total award and the online community created specifically for Cash for College 
Scholars in the Cash for College Performance-based Scholarship study. Once you claim your award and 
provide proof of your enrollment, you will receive your first payment at the start of your fall term. 

Save this letter and the attached information. 

You will need the ID in the box below to claim your award. 
SCHOLARSHIP CLAIM ID: <ID#> 


If you have any questions about this scholarship, send an e-mail to our scholarship office at cashforcol- 
lege@lachamber.com . 

If you have not already chosen the college you plan to attend in the fall, it’s not too late! Visit 
www.californiacolleges.edu or www.icanaffordcollege.com to find a college near you. 

Best wishes for your future success! 

Sincerely, 

California Cash for College 
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2010 CFC-PBS Study Claiming Instructions and Information 



HOW TO CLAIM YOUR CASH FOR COLLEGE PERFORMANCE-BASED STUDY SCHOLARSHIP: 


1) Beginning June 1, visit www.calgrants.org/cashforcollegescholarship to claim your Cash for College Perfor- 
mance-based Scholarship. On the website, you will be asked to enter your unique scholarship claiming identification 
number listed in your scholarship award letter, to update your contact information, to select the college or university 
you will attend this fall, and to list the date when you will enroll at your institution. You need to go the website to 
claim your award beginning Tuesday, June 1, 2010. You may lose your scholarships in whole or in part if it is 
not claimed by August 3 rd . If you do not have your scholarship award letter with your scholarship claim ID, you can 
email our scholarship office to get your scholarship claiming identification number at cashforcol- 
lege@lachamber.com . 

2) To receive your first payment, please send in your fall 2010 class schedule within two weeks of the date you 
enroll at your selected college or university. Your fall 2010 class schedule should include some obvious identifi- 
cation that it’s from your campus (e.g. University name, logo or URL) and the number of units for which you are 
enrolled. You may either FAX or mail your schedule to Carmen Gomez, Scholarship Administrator, 213-482-0814 
or 350 S. Bixel Street, Suite 200, Los Angeles, CA 90017. 

3) If you do not plan to attend college in the fall term, please contact our scholarship office at 
cashforcollege@lachamber.com . You may still be eligible to receive a portion of your scholarship if you begin 
school later in the 2010-2011 academic year. 

SCHOLARSHIP REQUIREMENTS 

You must meet the following requirements to receive an initial scholarship award: 

• Graduate from a California high school; 

• Enroll half-time (6 units) or more at an accredited, degree-granting, 2-year or 4-year post-secondary institu- 
tion in the fall of 2010. Your institution can be located anywhere in the United States. 

• Pursue either an Associate’s (2-year) Degree or a Bachelor’s (4-year) Degree. 

• Vocational schools or technical schools are not qualifying institutions for this scholarship. 

• Vocational courses which lead to an Associate’s degree at California Community Colleges do qualify for 
this scholarship. 

If you have questions about these requirements, please e-mail cashforcollege@lachamber.com. 

Note: 6 credit hours/units are based upon enrollment in a semester system. Equivalent requirements must be met for 
institutions on different calendars. 

ABOUT YOUR AWARD 


You have received a Cash for College Performance-Based Scholarship. Performance-based means that in order to 
receive your scholarship, you must maintain a 2.0 grade point average or better across all of your college courses for 
the semester/quarter; and complete at least 6 credit hours/units for the term. 

Payment of your Performance Based scholarship will be as follows: 

• Your scholarship will be sent to you in multiple payments. You won’t get the whole amount at once. 

• Your first payment is based on enrollment and will be sent to you at the start of the fall term, once 
verification of your enrollment in 6 credit hours/units for the term has been received. Once your 
enrollment verification is reviewed and approved, we will contact you to provide us with the necessary 
information to make an electronic funds transfer into your bank account. 
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• Future payments are based on achievement and will be made after the end of each term, once you have 
submitted verification that shows you have earned a 2.0 grade point average or better and completed at 
least 6 credit hours/units for the term. 

This may seem a bit confusing at first and that is why we have set up an on-line community to provide you 
with specific information about your scholarship, the payments, and other information for your success. 

THE CASH FOR COLLEGE PERFORMANCE-BASED SCHOLARSHIP STUDY ON-LINE 
COMMUNITY 


When you claim your scholarship, you will be directed to the Cash for College on-line community created for Cash 
for College Scholars who are part of the Cash for College Performance-based Scholarship study. You are a select 
group of students who will be attending colleges and universities across the US and this website was created 
specifically for you, with your success in mind. An individual webpage will be created for you to provide you with 
detailed information on the amount of each scholarship check, based upon the institution you plan to attend. You 
will also be able to track how much you have earned and access a variety of other important information. This 
website will be your primary source of information about your scholarship. You should bookmark the website 
address when you visit and visit it frequently. Should you lose the address, contact our scholarship office at 
cashforcollege@lachamber.com. 

ADDITIONAL INFORMATION 


Notification to the Financial Aid Office 

After you complete the online claim form, and your fall 2010 class schedule is received by the Cash for College 
Scholarship Office, notification of your award will be sent directly to your college or university’s financial aid 
office. You may then see your scholarship included on your Financial Aid Award Letter. The Financial Aid office is 
being asked to consider this award as additional assistance above and beyond grant aid. Should you have specific 
questions related to your financial aid package, you should always visit the Financial Aid Office at your institution. 

Providing Enrollment and Grade Verification 

To receive your scholarship, you must submit verification of your enrollment and proof that you have met the 
performance criteria for each term. Official documents showing that you have met the enrollment and performance 
criteria are preferred, but unofficial documents will also be accepted. 

We take the submission of these documents very seriously and a percentage of all unofficial documents will be 
selected for verification. Much like the Financial Aid process, if your documents are selected for verification, you 
will be asked to submit official documentation: an official transcript. Any student found to be submitting fraudulent 
documents may lose their scholarship. In addition, if we find there are an unusually high proportion of students 
misrepresenting their enrollment or grades, official transcripts may be required of all Cash for College Scholars in 
the Performance-based Scholarship study. 

Congratulations on your award! We wish you continued success with your education! 

Still have questions? If you have additional questions, please e-mail our scholarship office at cashforcol- 
lege@lachamber.com and don’t forget to visit the Cash for College Scholars website at: 
www.cashforcollegescholars.com . 

The California Cash for College Program is a partnership effort sponsored by the California Student Aid Commission, regional 
Cash for College coalitions, local high schools, community colleges, universities, outreach programs, community, local 
government and business groups. Visit www.californiacashforcollege.org for more infomiation. 
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The Performance-Based Scholarship Demonstration 
Appendix Figure B.2 

Sample CFC-PBS Control Group Notification Letter 
Cash for College Performance-Based Scholarship Study 


<Date> 

<Student First Name> <Student Last Name> 

<Street Address> 

<City>, <State> <Zip Code> 

Dear <Student First Name>, 

Thank you for attending a Cash for College Workshop earlier this year and for agreeing to participate in the Cash for 
College Performance Based Scholarship study being conducted by MDRC. As you may recall, there were a limited 
number of scholarships, and a lottery was used to select recipients. The lottery selection process has now been 
completed, and you were not one of those randomly chosen to receive a scholarship. Flowever, by attending a Cash 
for College workshop and submitting your FAFSA and Cal Grant GPA Verification, you took important steps to 
pursue higher education that will help you reach your personal and professional goals. While you were not selected 
to receive a scholarship, you did take steps towards completing the financial aid process which may make you 
eligible to receive the Federal and State aid available to all students. In addition, you may be eligible for other 
scholarships that are not part of this study. 

Although you were not selected to receive a scholarship, you still play an important role in the study as a member of 
the non-scholarship research group. Your continued participation in the study is critical to help us understand 
whether scholarships make a difference for students. Researchers will be in touch with you at some point in the 
future to ask you questions or administer a survey about your college experience. The answers you give may benefit 
others, and you will be compensated for your participation. Should you change your address during the next few 
years, please contact Michelle Ware at the phone number or email below, with your new contact information. 

On behalf of MDRC, I wish you the very best in pursuing your college plans. If you have any questions, please feel 
free to contact Michelle Ware at 1-800-221-3165 or by email at michelle.ware@mdrc.org . 

Thank you and best wishes for a successful college career. 
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Supplementary Exhibit for Chapter 3 




The Performance-Based Scholarship Demonstration 
Appendix Table C.l 
Quarter Payment Schedule 


Cash for College Performance-Based Scholarship Study 


Characteristic 

1 

2 

Scholarship Type 

3 4 

5 

6 

Performance-based scholarship 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Amount of scholarship per quarter ($) 

1,000 

1,000 

333 

667 

333 

667 

Duration of scholarship 

1 quarter 

1 quarter 

3 quarters 

3 quarters 

6 quarters 

6 quarters 

First year 
Fall payments ($) 
Enrollment 

1,000 

500 

167 

333 

167 

333 

Performance 

- 

500 

167 

333 

167 

333 

Winter payment ($) 
Performance 

— 

— 

333 

667 

333 

667 

Spring payment ($) 
Performance 

— 

— 

333 

667 

333 

667 

Second year 
Fall payments ($) 
Enrollment 





167 

333 

Performance 

- 

- 

- 

- 

167 

333 

Winter payment ($) 
Performance 

— 

— 


— 

333 

667 

Spring payment ($) 
Performance 

- 

- 

- 


333 

667 

Total scholarship amount ($) 

1,000 

1,000 

1,000 

2,000 

2,000 

4,000 


NOTE: Students who attended quarter institutions can earn the same dollar amount in aggregate as those who attended semester institutions, but 
payments were divided into three quarters of the academic year. The majority of the students in the CFC-PBS study attended semester-based 
institutions. 




Appendix D 

Supplementary Exhibits for Chapter 4 




The Performance-Based Scholarship Demonstration 
Appendix Table D.l 

Impact Matrix of Enrollment in First Term, by Scholarship Type 


Cash for College Performance-Based Scholarship Study 




$1,000 over 

$1,000 over 

$1,000 over 

$2,000 over 

$2,000 over 

$4,000 over 


Control 

1 Term, No 

1 Term, 

1 Year, 

1 Year, 

2 Years, 

2 Years, 


Group 

Performance 

Performance 

Performance 

Performance 

Performance 

Performance 

Characteristic (%) 

Mean 

Incentive 

Incentive 

Incentive 

Incentive 

Incentive 

Incentive 

Control group 

84.4 

3.5 

4.1 

* 3.9 

* 5.1 

** 5.3 




(2.1) 

(2.2) 

(2.2) 

(2.2) 

(2.2) 

(2.2) 

$1,000 over 1 term. 



0.6 

0.4 

1.6 

1.8 

2.4 

no performance incentive 

- 

- 

(3.0) 

(2.9) 

(2.9) 

(2.9) 

(2.9) 

$1,000 over 1 term, 




-0.2 

1.0 

1.2 

1.8 

performance incentive 

- 

- 

- 

(3.0) 

(3.0) 

(3.0) 

(3.0) 

$1,000 over 1 year, 





1.2 

1.3 

2.0 

performance incentive 

- 

- 

- 

- 

(2.9) 

(2.9) 

(3.0) 

$2,000 over 1 year, 






0.2 

0.8 

performance incentive 

- 

- 

- 

- 

- 

(2.9) 

(2.9) 

$2,000 over 2 years, 







0.6 

performance incentive 

- 

- 

- 

- 

- 

" 

(2.9) 

Sample size 








(total = 4,921) 

3,281 

279 

264 

273 

276 

276 

272 


(continued) 





Appendix Table D.l (continued) 


SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: To analyze whether scholarship type predicted first-semester registration, a joint test was performed. This test yielded a p- 
value for the F-statistic of less than 0.01, which is significant at the 1 percent level. This suggests that the differences in first-semester 
registration between scholarship types and the control group are unlikely to have occurred by chance. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

Standard errors are reported in parentheses under impact estimates. 

National Student Clearinghouse data were not found for 320 students (6.5 percent of the sample). 



The Performance-Based Scholarship Demonstration 
Appendix Table D.2 

Impact Matrix of Enrollment in Second Term, by Scholarship Type 


Cash for College Performance-Based Scholarship Study 




$1,000 over 

$1,000 over 

$1,000 over 

$2,000 over 

$2,000 over 

$4,000 over 


Control 

1 Term, No 

1 Term, 

1 Year, 

1 Year, 

2 Years, 

2 Y ears, 


Group 

Performance 

Performance 

Performance 

Performance 

Performance 

Performance 

Characteristic (%) 

Mean 

Incentive 

Incentive 

Incentive 

Incentive 

Incentive 

Incentive 

Control Group 

84.7 

2.8 

3.5 

5.6 

** 3.8 

* 2.7 

3.0 



(2.2) 

(2.2) 

(2.2) 

(2.2) 

(2.2) 

(2.2) 

$1,000 over 1 term. 



0.6 

2.7 

0.9 

-0.1 

0.2 

no performance incentive 

- 

- 

(3.0) 

(2.9) 

(2.9) 

(2.9) 

(2.9) 

$1,000 over 1 term, 




2.1 

0.3 

-0.7 

-0.4 

performance incentive 

- 

- 

- 

(3.0) 

(3.0) 

(3.0) 

(3.0) 

$1,000 over 1 year, 





-1.8 

-2.8 

-2.5 

performance incentive 

- 

- 

- 

- 

(2.9) 

(2.9) 

(3.0) 

$2,000 over 1 year, 






-1.1 

-0.8 

performance incentive 

- 

- 

- 

- 

- 

(2.9) 

(3.0) 

$2,000 over 2 years, 







0.3 

performance incentive 

- 

- 

- 

- 

- 

" 

(3.0) 

Sample size 








(total = 4,921) 

3,281 

279 

264 

273 

276 

276 

272 


(continued) 



Appendix Table D.2 (continued) 


SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: To analyze whether scholarship type predicted second-semester registration, a joint test was performed. This test yielded a 
value for the F-statistic of 0.03, which is significant at the 5 percent level. This suggests that the differences in second-semester 
registration between scholarship types and the control group are unlikely to have occurred by chance. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

Standard errors are reported in parentheses under impact estimates. 

National Student Clearinghouse data were not found for 320 students (6.5 percent of the sample). 



The Performance-Based Scholarship Demonstration 
Appendix Table D.3 

Enrollment Outcomes Among Students with One-Term Scholarships, 
Non-Performance-Based Versus Performance-Based: 

First Through Third Terms 


Cash for College Performance-Based Scholarship Study 



Average Outcome Levels 

Non-PBS 
vs. Control 

PBS 

vs. Control 

PBS 

vs. Non-PBS 

Outcome (%) 

Non- 

PBS 

PBS 

Control 

Group 

Standard 
Difference Error 

Standard 
Difference Error 

Standard 
Difference Error 

First term 

Enrolled in any college 3 

87.9 

88.5 

84.4 

3.5 

2.2 

4.1 * 

2.3 

0.6 

3.0 

Enrolled in any 2-year college 

48.3 

47.4 

43.2 

5.2 

* 3.1 

4.2 

3.1 

-1.0 

4.2 

Enrolled in a California community college 

48.0 

46.6 

42.7 

5.3 

* 3.1 

3.9 

3.1 

-1.4 

4.2 

Enrolled in any 4-year college 

39.9 

42.3 

42.8 

-2.9 

3.0 

-0.6 

3.1 

2.4 

4.2 

Enrolled in a California State University college 

22.6 

21.2 

22.9 

-0.3 

2.6 

-1.7 

2.7 

-1.4 

3.6 

Enrolled in a University of California college 

11.2 

14.3 

13.0 

-1.8 

2.1 

1.3 

2.1 

3.1 

2.9 

Second term 

Enrolled in any college 3 

87.5 

88.1 

84.7 

2.8 

2.2 

3.5 

2.3 

0.6 

3.0 

Enrolled in any 2-year college 

51.6 

51.9 

47.0 

4.5 

3.1 

4.9 

3.2 

0.4 

4.3 

Enrolled in a California community college 

51.2 

51.2 

46.4 

4.8 

3.1 

4.8 

3.2 

0.0 

4.3 

Enrolled in any 4-year college 

38.1 

41.1 

42.1 

-4.0 

3.0 

-1.0 

3.1 

3.0 

4.2 

Enrolled in a California State University college 

22.2 

20.8 

22.2 

0.0 

2.6 

-1.4 

2.6 

-1.5 

3.6 

Enrolled in a University of California college 

9.7 

14.3 

12.7 

-3.0 

2.1 

1.6 

2.1 

4.5 

2.8 


(continued) 



Appendix Table D.3 (continued) 


Outcome (%) 

Average Outcome Levels 

Non-PBS 
vs. Control 

PBS 

vs. Control 

PBS 

vs. Non-PBS 

Non- 

PBS 

PBS 

Control 

Group 

Standard 
Difference Error 

Standard 
Difference Error 

Standard 
Difference Error 

Third term 










Enrolled in any college 3 

80.0 

80.9 

79.0 

1.0 

2.5 

1.8 

2.6 

0.8 

3.4 

Enrolled in any 2-year college 

46.2 

45.0 

41.4 

4.8 

3.1 

3.6 

3.1 

-1.2 

4.2 

Enrolled in a California community college 

45.5 

43.9 

40.8 

4.7 

3.1 

3.1 

3.1 

-1.6 

4.2 

Enrolled in any 4-year college 

34.5 

37.0 

38.2 

-3.7 

3.0 

-1.2 

3.1 

2.5 

4.1 

Enrolled in a California State University college 

19.7 

18.2 

19.9 

-0.2 

2.5 

-1.8 

2.5 

-1.6 

3.4 

Enrolled in a University of California college 

9.0 

13.1 

11.7 

-2.7 

2.0 

1.4 

2.0 

4.1 

2.7 

Sample size (total = 3,824) 

279 

264 

3,281 








SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

Calculations for this table used all available data for the 3,824 control group, Scholarship Type 1, and Scholarship Type 2 sample members in 
the California study. 

National Student Clearinghouse data were not found for 268 students (7.0 percent of the sample). 
a A small proportion of students were enrolled at more than one institution. 





The Performance-Based Scholarship Demonstration 
Appendix Table D.4 

Enrollment Outcomes Among Students with One-Year Scholarships, $500 per Term Versus $1,000 per Term: 

First Through Third Terms 


Cash for College Performance-Based Scholarship Study 



Average Outcome Levels 

$500 per Term 

$1,000 per Term 

$1,000 per Term 


$1,000 

vs. Control 

vs. Control 

vs. $500 per Term 


$500 per per Control 

Standard 

Standard 

Standard 

Outcome (%) 

Term Term Group 

Difference Error 

Difference Error 

Difference Error 


First term 


Enrolled in any college 3 

88.3 

89.5 

84.4 

4.0 * 

2.2 

5.1 ** 

2.2 

1.2 

3.0 

Enrolled in any 2-year college 

49.0 

49.5 

43.2 

5.8 * 

3.1 

6.3 ** 

3.1 

0.6 

4.2 

Enrolled in a California community college 

49.0 

49.5 

42.7 

6.2 ** 

3.1 

6.8 ** 

3.1 

0.6 

4.2 

Enrolled in any 4-year college 

40.8 

41.4 

42.8 

-2.0 

3.1 

-1.4 

3.1 

0.6 

4.2 

Enrolled in a California State University college 

21.7 

21.1 

22.8 

-1.2 

2.6 

-1.8 

2.6 

-0.6 

3.6 

Enrolled in a University of California college 

11.9 

14.2 

12.9 

-1.1 

2.1 

1.3 

2.1 

2.3 

2.8 

Second term 

Enrolled in any college 3 

90.2 

88.4 

84.7 

5.6 ** 

2.2 

3.8 * 

2.2 

-1.8 

3.0 

Enrolled in any 2-year college 

53.8 

52.8 

47.0 

6.8 ** 

3.1 

5.8 * 

3.1 

-1.0 

4.2 

Enrolled in a California community college 

52.4 

52.8 

46.4 

6.0 * 

3.1 

6.4 ** 

3.1 

0.5 

4.2 

Enrolled in any 4-year college 

41.2 

40.3 

42.1 

-0.9 

3.1 

-1.7 

3.1 

-0.8 

4.2 

Enrolled in a California State University college 

22.0 

20.7 

22.2 

-0.2 

2.6 

-1.5 

2.6 

-1.3 

3.5 

Enrolled in a University of California college 

11.9 

14.2 

12.7 

-0.8 

2.1 

1.5 

2.1 

2.3 

2.8 


(continued) 


Appendix Table D.4 (continued) 



Average Outcome Levels 

$500 per Term 

$1,000 per Term 

$1,000 per Term 


$1,000 

vs. Control 

vs. Control 

vs. $500 per Term 


$500 per per Control 

Standard 

Standard 

Standard 

Outcome (%) 

Term Term Group 

Difference Error 

Difference Error 

Difference Error 


Third term 


Enrolled in any college 3 

82.2 

80.9 

79.0 

3.2 

2.5 

1.9 

2.5 

-1.4 

3.4 

Enrolled in any 2-year college 

45.9 

43.1 

41.4 

4.4 

3.1 

1.7 

3.1 

-2.8 

4.2 

Enrolled in a California community college 

44.8 

43.1 

40.8 

4.0 

3.1 

2.3 

3.1 

-1.7 

4.2 

Enrolled in any 4-year college 

36.7 

38.5 

38.2 

-1.4 

3.0 

0.3 

3.0 

1.8 

4.1 

Enrolled in a California State University college 

19.1 

19.3 

19.9 

-0.9 

2.5 

-0.7 

2.5 

0.2 

3.4 

Enrolled in a University of California college 

10.4 

13.1 

11.7 

-1.3 

2.0 

1.4 

2.0 

2.7 

2.7 

Sample size (total = 3,830) 

273 

276 

3,281 








SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

Calculations for this table used all available data for the 3,830 control group. Scholarship Type 3, and Scholarship Type 4 sample members in 
the California study. 

National Student Clearinghouse data were not found for 272 students (7.1 percent of the sample). 
a A small proportion of students were enrolled at more than one institution. 


The Performance-Based Scholarship Demonstration 
Appendix Table D.5 

Enrollment Outcomes Among Students with Two-Year Scholarships, $500 per Term Versus $1,000 per Term: 

First Through Third Terms 


Cash for College Performance-Based Scholarship Study 


Outcome (%) 

Average Outcome Levels 
$1,000 

$500 per per Control 

Term Term Group 

$500 per Term 
vs. Control 
Standard 
Difference Error 

$1,000 per Term 
vs. Control 
Standard 
Difference Error 

$1,000 per Term 
vs. $500 per Term 
Standard 
Difference Error 

First term 

Enrolled in any college 2 

89.7 

90.3 

84.4 

5.3 

** 2.2 

5 g *** 

2.2 

0.6 

3.0 

Enrolled in any 2-year college 

43.9 

49.6 

43.1 

0.8 

3.1 

6.5 ** 

3.1 

5.7 

4.2 

Enrolled in a California community college 

43.2 

49.3 

42.7 

0.5 

3.1 

6.5 ** 

3.1 

6.0 

4.2 

Enrolled in any 4-year college 

47.1 

42.5 

42.8 

4.3 

3.1 

-0.3 

3.1 

-4.6 

4.2 

Enrolled in a California State University college 

25.0 

26.5 

22.9 

2.1 

2.6 

3.6 

2.7 

1.5 

3.6 

Enrolled in a University of California college 

16.0 

10.2 

13.0 

3.0 

2.1 

-2.8 

2.1 

-5.8 ** 

2.8 

Second term 

Enrolled in any college 2 

87.4 

87.7 

84.7 

2.7 

2.2 

3.0 

2.2 

0.3 

3.0 

Enrolled in any 2-year college 

47.9 

50.6 

47.0 

0.9 

3.1 

3.6 

3.1 

2.8 

4.2 

Enrolled in a California community college 

47.2 

50.3 

46.4 

0.8 

3.1 

3.9 

3.1 

3.1 

4.2 

Enrolled in any 4-year college 

44.6 

41.1 

42.1 

2.5 

3.1 

-1.0 

3.1 

-3.5 

4.2 

Enrolled in a California State University college 

22.1 

25.7 

22.2 

-0.1 

2.6 

3.5 

2.6 

3.6 

3.6 

Enrolled in a University of California college 

16.0 

9.8 

12.7 

3.2 

2.1 

-2.9 

2.1 

-6.2 ** 

2.8 


(continued) 





Appendix Table D.5 (continued) 


Average Outcome Levels $500 per Term $1,000 per Term $1,000 per Term 
$1,000 vs. Control vs. Control vs. $500 per Term 

$500 per per Control Standard Standard Standard 

Outcome (%) Term Term Group Difference Error Difference Error Difference Error 

Third term 


Enrolled in any college 3 

83.4 

79.5 

79.0 

4.4 * 

2.5 

0.5 

2.5 

-3.9 

3.4 

Enrolled in any 2-year college 

42.8 

42.9 

41.4 

1.4 

3.1 

1.4 

3.1 

0.1 

4.2 

Enrolled in a California community college 

41.7 

41.0 

40.8 

0.9 

3.1 

0.2 

3.1 

-0.7 

4.2 

Enrolled in any 4-year college 

42.1 

38.1 

38.2 

3.9 

3.0 

-0.1 

3.0 

-3.9 

4.1 

Enrolled in a California State University college 

21.0 

25.0 

19.9 

1.1 

2.5 

5.1 ** 

2.5 

4.0 

3.4 

Enrolled in a University of California college 

15.2 

8.0 

11.7 

3.5 * 

2.0 

-3.8 * 

2.0 


2.7 

Sample size (total = 3,829) 

276 

272 

3,281 








SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. 

Estimates are adjusted by research cohort and workshop region. 

Rounding may cause slight discrepancies in sums and differences. 

Calculations for this table used all available data for the 3,829 control group, Scholarship Type 5, and Scholarship Type 6 sample members in 
the California study. 

National Student Clearinghouse data were not found for 270 students (7.1 percent) of the sample. 
a A small proportion of students were enrolled at more than one institution. 
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Supplementary Information and Exhibits for Chapter 5 




This appendix describes the marginal cost assumptions used in the analysis presented in Chapter 
5 in more detail. As Chapter 5 notes, the cost of program administration for the California Cash 
for College Performance-Based Scholarship (CFC-PBS) Program was estimated using program 
expenditure data as recorded by the Los Angeles Area Chamber of Commerce. Because all 
scholarship types were administered together, program expenditures were not broken down by 
scholarship type. As a result, in order to estimate the cost of administration for each individual 
scholarship type, it was necessary to develop a set of marginal cost assumptions. As discussed 
in Box 5.2, three marginal cost types were used in this analysis: 

• Marginal cost per scholarship offered. Assumes that the primary driver of 
cost is the number of scholarships offered, or the number of participants. 

• Marginal cost per scholarship semester. Assumes that the primary driver 
of cost is the number of scholarship semesters, or the length of the scholar- 
ship. 

• Marginal cost per potential payment. Assumes that the primary driver of 
cost is the number of potential payments, or the disbursement points. 

Appendix Table E.l shows the marginal cost assumptions that were used in this analysis. To 
estimate the administrative costs of each CFC-PBS scholarship type, the following steps were 
carried out: 

1. Assign a marginal cost type or types to each spending category. Each 
spending category was assigned the most appropriate marginal cost type in 
consultation with program staff. For example, the first row of the second 
panel of Appendix Table E.l shows that student tracking-related costs asso- 
ciated with the employment of a web developer were believed to be driven 
by the number of scholarships offered, or the number of students participat- 
ing in the program. However, some administrative spending categories 
seemed to be driven by more than one factor. For instance, personnel costs 
were most likely related to the number of scholarships offered, the number of 
scholarship semesters, and the number of potential payments. Therefore, as 
shown in the first panel of Appendix Table E.l, one-third of the amount 
spent on personnel is determined by the number of scholarships offered, an- 
other third is determined by the number of scholarship semesters, and the last 
third is determined by the number of potential payments. 

2. Determine the number of units during the analysis period. The expendi- 
ture infonnation used to carry out this analysis covered the period of January 
2009 through June 2011. For each marginal cost type listed above, the total 
number of units during that period was calculated. In particular, there were 
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The Performance-Based Scholarship Demonstration 
Appendix Table E.l 
Examining Marginal Cost Estimates 


Cash for College Performance-Based Scholarship Study 





Units 


Category 

Total 
Cost ($) Over 
2 Years 

Marginal 

Cost 

Category 

During 

2-Year 

Period 

Units by Type Over Cost Over 

Marginal Life of Scholarship Life of Scholarship ($) 

Cost ($) Type 1 Type 2 Type 3 Type 4 Type 1 Type 2 Type 3 Type 4 


Personnel 


Chamber of Commerce 

94,002 

Scholarship offer 

1,720 

54.65 

296 

277 

290 

285 

16,177 

15,139 

15,849 

15,576 

Chamber of Commerce 

94,002 

Scholarship semester 

3,365 

27.94 

296 

277 

580 

570 

8,269 

7,738 

16,202 

15,923 

Chamber of Commerce 

94,002 

Potential payment 

5,038 

18.66 

296 

554 

870 

855 

5,523 

10,337 

16,233 

15,953 

Personnel subtotal 









29,969 

33,214 

48,285 

47,452 


Student tracking 


Web developer 

21,632 

Scholarship offer 

1,720 

Data entry services 

20,644 

Scholarship offer 

1,720 

Data entry services 

20,644 

Potential payment 

5,038 

Research services 

28,170 

Scholarship offer 

1,720 

Research services 

28,170 

Potential payment 

5,038 


Student tracking subtotal 


12.58 

296 

277 

290 

285 

3,723 

3,484 

3,647 

3,584 

12.00 

296 

277 

290 

285 

3,553 

3,325 

3,481 

3,421 

4.10 

296 

554 

870 

855 

1,213 

2,270 

3,565 

3,503 

16.38 

296 

277 

290 

285 

4,848 

4,537 

4,750 

4,668 

5.59 

296 

554 

870 

855 

1,655 

3,098 

4,865 

4,781 






14,991 

16,713 

20,307 

19,957 


Communication and outreach 


General communication 

Statewide outreach 

16,845 

Scholarship offer 

1,720 

Statewide outreach 

16,845 

Potential payment 

5,038 

Website 

Maintenance costs 

69,470 

Scholarship semester 

3,365 

Web developer 

1,139 

Scholarship semester 

3,365 

Ceremonies 3 

11,313 

Scholarship offer 

1,720 


Communication subtotal 


9.79 

296 

277 

290 

285 

2,899 

2,713 

2,840 

2,791 

3.34 

296 

554 

870 

855 

990 

1,852 

2,909 

2,859 

20.64 

296 

277 

580 

570 

6,111 

5,719 

11,974 

11,768 

0.34 

296 

277 

580 

570 

100 

94 

196 

193 

6.58 

296 

277 

290 

285 

1,947 

1,822 

1,907 

1,875 






12,047 

12,200 

19,827 

19,485 
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Appendix Table E.l (continued) 





Units 


Category 

Total 
Cost ($) Over 
2 Y ears 

Marginal 

Cost 

Category 

During 

2-Year 

Period 

Units by Type Over CostOver 

Marginal Life of Scholarship Life of Scholarship ($) 

Cost ($) Type 1 Type 2 Type 3 Type 4 Type 1 Type 2 Type 3 Type 4 


Workshop and regional 
support 

L.A. CFC regional training 


Trainer fees 

15,000 

Scholarship offer 

1,720 

8.72 

296 

277 

290 

285 

2,581 

2,416 

2,529 

2,485 

Materials/administrative 












fees support 

10,778 

Scholarship offer 

1,720 

6.27 

296 

277 

290 

285 

1,855 

1,736 

1,817 

1,786 

Communication costs 
Facility rental and 

3,415 

Scholarship offer 

1,720 

1.99 

296 

277 

290 

285 

588 

550 

576 

566 

equipment 

2,250 

Scholarship offer 

1,720 

1.31 

296 

277 

290 

285 

387 

362 

379 

373 

Regional CFC program 
costs 













Region support ad- 













ministrative stipend 

57,700 

Scholarship offer 

1,720 

33.55 

296 

277 

290 

285 

9,930 

9,292 

9,728 

9,561 

Regional workshops 

4,038 

Scholarship offer 

1,720 

2.35 

296 

277 

290 

285 

695 

650 

681 

669 

Workshop and regional support subtotal 








16,036 

15,006 

15,711 

15,440 

Total* 3 ($) 

610,061 








73,043 

77,133 

104,130 102,334 

Sample size c 

Total per program group 

1,720 








296 

277 

290 

285 

member (Types 1-6) ($) 

355 








247 

278 

359 

359 
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Appendix Table E.l (continued) 


SOURCE: MDRC calculations from expenditure data provided to MDRC by the Los Angeles Chamber Foundation. 

NOTES: Program costs are based on a steady state of operation that excludes research and start-up costs. 

The expenditure information used to create this table covers the period of January 2009 through June 2011. The “2 -Year Period” column head 
refers to January 2009 through June 2011. 

Marginal costs are allocated in a way that treats all schools as semester schools. 

a The “ceremonies” cost category represents scholarship recognition ceremonies that were held in some regions for scholarship recipients. 
b The total two-year cost includes spending associated with Scholarship Types 1 through 6. The units during the two-year period describe the 
number of units from Scholarship Types 1 through 6. Since Scholarship Types 5 and 6 had a longer duration, this two-year observation period does 
not include all the expenditures associated with those two scholarship types. As a result, Scholarship Types 5 and 6 have been excluded from this 
analysis. 

c The sample size in the two-year cost column is the number of individuals receiving Scholarship Types 1 through 6. The cost of the individuals in 
Scholarship Types 5 and 6 will be estimated at a later date when complete expenditure data are available. 
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1,720 scholarships offered, 5,038 potential payments, and 3,365 scholarship 
semesters. 1 These totals include Scholarship Types 5 and 6, although they are 
not shown in Appendix Table E.l. Even though these two scholarship types 
are not included in the analysis discussed in Chapter 5 since they had not 
been fully paid out by the end of the analysis period, they were in operation 
during this period, and some of the total program cost was related to their 
administration. 

3. Calculate the marginal cost per unit. After assigning the most appropriate 
marginal cost type for each spending category and identifying the total num- 
ber of units during the analysis period for each marginal cost type, a marginal 
cost per unit was calculated for each spending category. As can be seen in 
Appendix Table E.l, the marginal cost per unit is equal to the total cost dur- 
ing the analysis period (“Two-Year Cost” shown in the second column) di- 
vided by the total number of units during the same period. For example, the 
marginal cost per scholarship offer of student tracking-related web developer 
costs (shown in the first row of the second panel) was $12.58 ($21,632 
1,720). 

4. Estimate the total cost for each scholarship type for every spending cat- 
egory. The marginal cost per unit was used to estimate the total cost associ- 
ated with each of Scholarship Types 1 through 4, by multiplying the cost per 
unit by the total number of the appropriate units during the analysis period 
for each scholarship type. Returning to the example of the student tracking- 
related web developer costs, to get the total cost over the two-year period for 
Scholarship Type 1, the marginal cost per scholarship offer was multiplied 
by the total number of scholarship offers, yielding $3,723 ($12.58 x 296). 

5. Calculate the cost per student for each scholarship type. After estimating 
the total cost for each scholarship type for every spending category, the total 
cost for each scholarship type (the sum of the total cost for all spending cate- 
gories) was divided by the number of program group members receiving that 
scholarship type to estimate the total administrative cost per program group 

Tor each scholarship type, the total number of potential payments is equal to the number of scholarship of- 
fers multiplied by the number of potential payments during the analysis period. For example, 277 students were 
offered Scholarship Type 2, which had two potential payments. As a result, the total number of potential 
payments for Scholarship Type 2 is 554 (277 x 2). Similarly, the number of scholarship semesters is the number 
of scholarship offers by the length of that individual scholarship type during the analysis period. For example, 290 
students were offered Scholarship Type 3, which lasted for two semesters in total, both of which took place during 
the analysis period, while for the 164 students who began in fall 2010, only two of them did. Therefore, the 
number of scholarship semesters for that scholarship type is 580 (290 x 2). 
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member, by scholarship type. This is shown in the rightmost four columns of 
the final row of Appendix Table E. 1 . 


Sample Size 

The sample size used to calculate the administrative cost per student (1,720 across the six 
scholarship types) differs from the sample used in the analysis of the program’s impacts (1,640 
across the six scholarship types). This is due to the fact that 239 students who were randomly 
assigned to either the program or control group were excluded from the impact analysis as a 
result of incomplete data collection, as noted in Chapter 2. These students were included in the 
calculation of administrative cost per student because even though their outcomes were not 
analyzed, the program still incurred costs related to their participation. Therefore, excluding 
them would have made the cost per program group member look artificially high. 
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The Performance-Based Scholarship Demonstration 
Appendix Table F.l 

Descriptions of Prior External Incentive Research 


Cash for College Performance-Based Scholarship Study 


Study 

Eligible Population 

Need- 

Based 

Number 
of Sites 

Program 

Groups 

Incentive 

Duration 

Maximum 

Incentive 

Amount 

Academic 

Benchmarks 

Additional 

Service 

Criteria 

Social Security 
Student Benefit 
Program 
(United States) 
1979-1983 

High school seniors with 
parent who is a Social 
Security beneficiary 3 

Yes 

National 

1 

8 

semesters 

$6,700 per 

b 

year 

Benefits based 
on family 
earnings and 
continued 
enrollment 

None 

Cal Grant program 
(United States) 
1999-2000 

lst-year college students 
who applied for financial 
aid 

Yes 

Statewide 

2 

2 

semesters 

Cal Grant 
A: $3,429 

Cal Grant 
B: $l,409 c 

Cal Grant 
eligibility: 
income, assets, 
and high school 
GPA 

requirements 

None 

Project STAR 

(Canada) 

2005-2006 

Full-time, lst-year 
college students with high 
school GPAs in 1st to 3rd 
quartile d 

No 

1 

3 

2 

semesters 

$5,000 

Meet GPA 
benchmark 
depending on 
high school 
quartile 

Register for 
second year 

Peer 

advising 0 


(continued) 




















Appendix Table F.l (continued) 


Study 

Eligible Population 

Need- 

Based 

Number 
of Sites 

Program 

Groups 

Incentive 

Duration 

Maximum 

Incentive 

Amount 

Academic 

Benchmarks 

Additional 

Service 

Criteria 

University of 
Amsterdam 
(Netherlands) 
2001-2002 

lst-year college students 
in economics and 
business 

No 

1 

2 

3 terms 

$215 or 
$644 f 

Complete all lst- 
year 

requirements by 
start of next 
term 

None 

West Virginia 
PROMISE 
(United States) 
2000-2004 

Full-time college students 
with minimum high 
school GPA and ACT or 
SAT scores 

No 

Statewide 

1 

8 

semesters 

$10,000 g 

Meet 3.0 
cumulative GPA 
benchmark 

Complete at 
least 30 credits 
per year 

None 

Wisconsin 
Scholars 
(United States) 
2008-2011 

lst-year college students 
who are Pell-eligible 

Yes 

13 

1 

10 

semesters 

$17,500 

Maintain Pell 
eligibility 1 * 

Register for at 
least 12 credits 
per term 

None 

Opportunity 

Knocks 

(Canada) 

2008-2009 

1st- and 2nd-year college 
students who applied for 
financial aid, have a high 
school GPA on file, and 
enrolled for half of a full 
load 

No 

1 

1 

2 

semesters 

$7,000* 

Attain minimum 
grade of 70 
(benchmark by 
course) 

Peer 

advising 
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Appendix Table F.l (continued) 


Study 

Eligible Population 

Need- 

Based 

Number 
of Sites 

Program 

Groups 

Incentive 

Duration 

Maximum 

Incentive 

Amount 

Academic 

Benchmarks 

Additional 

Service 

Criteria 

Foundations for 
Success 
(Canada) 
2007-2009 

Full-time students who 
are Canadian citizens and 
identified as at risk based 
on Accuplacer and 
FastTrack surveys 

No 

3 

2 

3 

semesters 

$2,250 

Meet 2.0 GPA 
benchmark 

Be eligible to 
continue at 
institution 

12 hours of 
activity 

Opening Doors 
Louisiana 
(United States) 
2004-2005 

Parents age 18 to 34 with 
family incomes below 
200 percent of federal 
poverty level 

Yes 

2 

1 

2 

semesters 

$2,000 

Complete 6 or 
more credits 
with a “C” 
average or better 

Meet with 
adviser 

PBS California 
(United States) 
2009-2012 

High school seniors age 
16 to 19 applying for 
financial aid who are 
below Cal Grant A/C 
income threshold J 

Yes 

Portable 

5 

1 term to 2 
years 

$1,000 to 
$4,000 

Complete 6 or 
more credits 
with a “C” 
average or better 

None 
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Appendix Table F.l (continued) 


Study 

Eligible Population 

Need- 

based 

Number 
of sites 

Program 

groups 

Incentive 

Duration 

Maximum 

Incentive 

Amount 

Academic 

Benchmarks 

Additional 

Service 

Criteria 

PBS New Mexico 
(United States) 
2008-2011 

Freshmen age 17 to 20 
who are Pell-eligible 

Yes 

1 

1 

4 

semesters 

$4,000 

Complete 12 or 
more credits (1st 
semester) or 15 
credits 
(subsequent 
semesters) with 
a “C” average or 
better 

Meet with 
adviser 

PBS New York 
(United States) 
2008-2010 

Students age 22 to 35 
who are living away from 
parents, are in need of 
developmental education, 
and are Pell-eligible 

Yes 

2 

2 

2 full 

semesters 
and 1 
summer 
semester k 

$2,600 or 
$3,900 

Complete 6 or 
more credits 
with “C” or 
better in each 

None 

PBS Ohio 
(United States) 
2008-2010 

Parents age 18+ with a 
zero EFC 1 

Yes 

3 

1 

2 

semesters 
or 3 

quarters 

$1,800 

Part-time: 
Complete 6 to 
1 1 credits with 
“C” or better in 
each 

Full-time: 
Complete 12 or 
more credits 
with “C” or 
better in each 

None 
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Appendix Table F.l (continued) 


SOURCES: Social Security Student Benefit Program (Dynarski, 2003); Cal Grant Program (Kane, 2003); Project STAR (Angrist, 
Lang, and Oreopoulos, 2009); University of Amsterdam (Leuven, Oosterbeek, and van der Klaauw, 2010); West Virginia PROMISE 
(Scott-Clayton, 2011); Wisconsin Scholars (Goldrick-Rab, Harris, Benson, and Kelchen, 2011); Opportunity Knocks (Angrist, 
Oreopoulos, and Williams, 2010); Foundations for Success (MacDonald et al., 2009); and MDRC Opening Doors Louisiana and PBS 
Demonstration studies. 

NOTES: a The empirical analysis limits the analysis group to those potentially eligible for Social Security benefits due to the death of 
a parent. 

b $6,700 is the average Social Security Student Benefit paid in 1980 to the children of deceased parents. 

c For Cal Grant A, students were eligible for up to $3,429 per year in 1999-2000 for tuition and required fees at a University of 
California or California State University school, or up to $9,420 for a private, four-year institution. For Cal Grant B, students were 
eligible for $1,409 per year. 

d Academic benchmark was conditional on high school grade quartiles. See Angrist, Lang, and Oreopoulos (2009) for description. 

e There were three program groups: The Student Support Program (SSP) group was eligible for peer advising, the Student 
Fellowship Program (SFP) group was eligible for incentive grants, and the SFSP group was eligible for both peer advising and the 
incentive grant. SFSP students did not need to participate in peer advising in order to earn the incentive grant. 

r The low-reward group was offered 227 euros ($215 in 2002 dollars); the high-reward group was offered 681 euros ($644 in 2002 
dollars). 

=The grant covers full tuition and required fees. Those who initially qualified received an average of about $ 1 0,000 over four 
years. As of 2010, the award was capped at $4,750 per student per year, but no maximum dollar amount was available for the study 
sample in the paper. 

h The Pell Grant requires that students make satisfactory academic progress (SAP), which typically means a “C” average or 
equivalent and “academic standing consistent with the requirements for graduation” from the institution (see 
https://studentaid.ed.gov/eligibility/staying-eligible). Apart from SAP, there were no stated GPA requirements for the Wisconsin 
Scholars Grant. 

‘The incentive grant is $100 per course plus an additional $20 per percentage point. A full course load consists of 10 credits per 
year. 

■>To qualify for Cal Grant financial aid, students must fall below certain income and asset ceilings. 

k The study in New York randomly assigned program group members to one of two scholarship types. One type was offered over 
two semesters only; the other was offered over two semesters plus one summer semester. 

'EFC, or Expected Family Contribution, is the amount of money a student is expected to either pay out-of-pocket or procure in 
loans to cover the costs associated with postsecondary attendance. All else being equal, a lower EFC is associated with higher levels 
of need-based aid. 
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and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
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Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
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• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



